MBCook · 9 years ago

But isn't that really what lets them do this? In following PyPy (and other alternate Python discussions) it seems like eliminating the GIL and getting better performance out of Python, even in C, isn't that hard if you drop the C extensions.

As soon as something can see into the guts of the interpreter you have to maintain compatibility which is a pain/waste.

Worse than that is that view wasn't designed for multi-threading which is why the GIL exists. The C extensions were t designed to be multi-threaded because that wasn't a thing in Python so they're not safe. You either have to drop them, define a new interface layer that would be safe, or I suppose somehow sandbox their little view of the world but keep it coherent between threads.

If you have a codebase where you can make the choice to drop C extensions and you're trying to accelerate Python it seems like a very smart choice.

sgift · 9 years ago

From the post it looks like it is optimized for multithreading from the start, so they possibly used the multithreaded version with thread count = 1. When you care for parallel workloads its not so important if your single-threaded code is the fastest.

machbio · 9 years ago

Guido insists that single threaded performance of Python has the highest priority - http://www.artima.com/weblogs/viewpost.jsp?thread=214235

I am not sure, any implementation of Python will beat the single threaded performance of Cpython..

trotterdylan · 9 years ago

Although Grumpy is compiled, it is just as dynamic as Python, in that method dispatch involves dictionary lookups, etc.

The main reason why Grumpy's slower for most single threaded benchmarks is that most Python workloads involve creating and freeing a bunch of small Python objects. In Go, these objects are garbage that need to be GC'd in a very general way. In CPython, there are free lists, arenas and other optimizations for allocating small (especially immutable) objects. And cleaning up garbage in CPython involves pushing unreferenced objects back onto the free lists for later reuse.

matt_wulfeck · 9 years ago

That's a question I have as well. I'm guessing they have a very massive cpython codebase, and the trade-off was worth it.

greenhatman · 9 years ago

Maybe there is too much to rewrite.

ericfrederich · 9 years ago

It's not just a matter of changing the implementation language from C to Go, they wanted to remove the GIL as well. For that you can't even use the existing CPython design.

jerf · 9 years ago

Ask yourself how much Python code you think Google has. Then multiply it by some large single-digit number, minimum. Then compare the effort of spending 30 seconds per line on that code vs. writing a new Python interpreter/compiler/runtime, especially when you can trivially get people who are capable of doing that.

There's a reason why the large companies often end up working on new runtimes/interpreters/compilers like HipHop for PHP, Hack, and so on, rather than working on the code bases written in those languages. It is very easy for it to not just be easier to leverage in at that level, but an order of magnitude or two easier. Or three.

eva1984 · 9 years ago

It could be part of the auto migration effort.

spankalee · 9 years ago

Because Python running in Grumpy can import Go packages, I'd assume (I work for Google, but not on YouTube), that Python libraries could then be migrated pretty seamlessly and incrementally to Go.

dragonwriter · 9 years ago

Creating a new Python runtime in Go is obviously non-trivial, but so is rewriting the heart of YouTube in Go. Heck, YouTube may be a bigger codebase than Python for all I know.

And there's a lot more side benefit in a better multithreading Python runtime for Google for other Python code Google has (or hosts), whereas the benefits of a YouTube rewrite are more narrowly limited to YouTube.

trotterdylan · 9 years ago

It's been said a couple times in this thread already but basically, the YouTube Python codebase is very large and the cost of a rewrite is prohibitive.

hueving · 9 years ago

I take it you've never worked on a large code base with a large team? Even ignoring the actual task of rewriting the logic in a different language with different idioms, the change is going to be a massive hit to productivity for many months as people learn the new architecture of the rewrite, the tooling, and idiomatic Go.

For a project the size of YouTube, that will be millions of dollars of engineering hours and weeks/months of lost productivity for an unknown gain and almost guaranteed bugs. It's a terrible value proposition so it's better to squeeze every last drop of performance out of the code base you have, which at the scale if this project includes paying engineer(s) to work on a completely new runtime.

petters · 9 years ago

The thing is, you can not use Python in production at Google. YouTube was acquired a long time ago and a rewrite of the inherited Pyton code base has not happened for very good reasons.

the_duke · 9 years ago

Well, if Google is pushing it and actually using it internally...

They also have the benefit of being able to push features into the Go core that they might need for this.

curtis · 9 years ago

There's no guarantee that it will succeed, but the fact that they're willing to sacrifice some considerable degree of compatibility to achieve other important goals may make the project a lot easier to pull off.

I still think Unladen Swallow should have been based on V8, but as I recall that project had very strict compatibility goals that would have made a V8-based implementation impossible.

takeda · 9 years ago

To me it looks like Google essentially is moving everything to Go, and this project is to help with it.

They port their python libraries so they can reference them from their Go code, and then module by module will rewrite it in pure Go.

scriptkiddy · 9 years ago

I made an agreement to myself to never use any Google language, tool, or library after they decided to shut down google code. Google has a history of creating all of these crazy projects and then just completely abandoning them leaving the early adopters to fend for themselves. It's really bad practice and leaves a nasty taste in my mouth. That's why I haven't even tried Go. I have no interest in it. As soon as Google develops some other language they'll drop Go like a hot potato and everyone using it in production will be forced to rewrite or stagnate.

Look at AngularJs for a more recent example.

Add to this that Google has some of the worst versioning practices I've ever seen and you get a recipe for destruction.

MBCook · 9 years ago

I wonder if they intend to maintain it long term or if they're just going to use it as a bridge while they rewrite things natively in Go.

trotterdylan · 9 years ago

I'd like to support 3.x at some point. See https://github.com/google/grumpy/issues/1

the_duke · 9 years ago

Actually, I would interpret this as "CPython is broken for scalability. It's not worth trying to fork/fix it, we'd rather just roll our own runtime".

Not exactly an endorsement.

Sadly, the code was just dumped into a new Git repo, so no way to tell how many people contributed internally so far.

Zikes · 9 years ago

I'm thinking it isn't in production yet, based on wording like "we're excited about the prospects" and "although it's still alpha software".

hacker_9 · 9 years ago

Yeah without performance metrics I'm not sure what to think about this. I'm surprised they even used Python to begin with for their front-end server, for such high volumes of traffic.

trotterdylan · 9 years ago

As has been said by others, Grumpy is not used in production at Google currently. There's still a lot of work to do -- especially on the standard library -- to support large real world codebases.

dijit · 9 years ago

cpython is the reference implementation, so it makes sense that it's;

A) Not well optimised.

B) Touting features before the spec/standard.

EDIT: people really dislike that I said this, and I'm having trouble finding my original citation- it was on one of the many python books I own. Most likely "Learn Python The Hard Way" but I'll dig out the exact chapter where they compare pypy to cpython and mention that because cpython is the reference implementation it values code clarity over performance optimisation.

devy · 9 years ago

> Basically, the language doesn't have a "spec" per-se.

It does[1]. And process of improving it is called PEP[2].

[1]: https://docs.python.org/

[2]: https://en.wikipedia.org/wiki/Python_(programming_language)#...

lobster_johnson · 9 years ago

The reference [1] is a decent spec. It might not be as formally rigorous as an ISO standard, but it's probably as good as Go's [2], which also a "reference".

[1] https://docs.python.org/3/reference/index.html

[2] https://golang.org/ref/spec

vram22 · 9 years ago

Does anyone know the technical reasons why C extensions are not (at least easily, apparently) supported by Go? Is it to do with Go's being a GC'd language? I would have thought that should not be a reason per se, since Python also has GC, but has plenty of extensions written in C. But I'm not a language internals expert.

Also, further signs that GC may not be the reason, is that D also has GC, but can link to C libraries somewhat easily (not sure about all cases or how far the ease goes).

trotterdylan · 9 years ago

I'm not philosophically opposed to supporting C extensions. The additional complexity just was not deemed to be warranted since the YouTube frontend doesn't use a lot of C extensions.

In principle it's possible to implement something like JyNI (http://jyni.org/) or CPyExt (https://morepypy.blogspot.de/2010/04/using-cpython-extension...) to bridge the CPython and Grumpy APIs. In practice, marshalling data across the interface can be very expensive.

lelandbatey · 9 years ago

I "grew up" on Python, then wrote a whole bunch of Go for my job. Then this past Autumn re-visited Python to implement a networked terminal based game[1].

With what I learned about Go and concurrency, I would say that currently in Python, writing concurrent code is not very hard, and is as close to Go as you can get without actually just writing Go.

Now, you may be saying "but Python has the GIL, how can concurrency be easy in Python?" I'd say, you're definitely not wrong that the GIL is a problem, but it's not much of a problem for concurrency.

This goes back to the heart of Rob Pike's classic talk, "Concurrency Is Not Parallelism"[2]. To quote Wikipedia:

        In computer science, concurrency is the decomposability property of a
        program, algorithm, or problem into order-independent or partially-ordered
        components or units.

In closing, concurrency in Python is pretty easy to work with, it just performs very poorly.

[1] - https://github.com/lelandbatey/defuse_division

[2] - https://blog.golang.org/concurrency-is-not-parallelism

pjmlp · 9 years ago

I never saw Python as anything more than a scripting language to portably automate tasks across UNIX and Windows environment, even back in the Zope days.

My experience with Tcl teached me to stay away from languages that don't have either a JIT or AOT compiler on their reference implementation.

jerf · 9 years ago

A huge part of the reason Python (and Perl, PHP, etc. in their original interpreters) is so slow is that it is like running a Go program that does everything through runtime reflection, even just to add two integers. If your code is already in Python, this level of performance is apparently not a problem for you.

If you know Go or are willing to learn about Go and reflection, you can learn a lot about how dynamic languages work under the hood by implementing:

    func Add(interface{}, interface{}) interface{} { ... }

trotterdylan · 9 years ago

Reflection in Go is used minimally in Grumpy because it is slow. Currently importing Go packages into Python code is accomplished via the reflect module, but I think this will have to change to make such integration useful.

ericfrederich · 9 years ago

I hope this is a well thought out solution that can evolve into something great... and not just something built for a single purpose.

I question the transpiler. I think I'd much rather prefer a solution like Jython.

gigatexal · 9 years ago

I'm sure as this gets hacked on they'll be able to support consuming imports and doing all the conversion to go recursively

trotterdylan · 9 years ago

Thanks for trying it out! And sorry about the lacking documentation. I'll be fleshing it out over the next little while.

Your assessment is right: the grumpc compiler takes a single Python file and spits out a Go package. Incidentally, this means you can import a Python module into Go code pretty easily.

I don't have a ready solution for building a large existing project but I'll write up a quick doc to outline the process. The trickiest bit is that the Python statement "import foo.bar" translates to a Go import: import "prefix/foo/bar". Currently prefix always points at the grumpc/lib directory so that's one way to integrate your code, but I need to make it more configurable.

brettcannon · 9 years ago

The decimal module in Python 2.7 is implemented in pure Python: https://github.com/python/cpython/blob/2.7/Lib/decimal.py (in Python 3 there is an accelerated extension module that's used when available, falling back on the pure Python version otherwise).

ot · 9 years ago

The objective of the authors is to efficiently exploit multi-core parallelism. PyPy still has a GIL. They have been doing some experiments with transactional memory, but performance is quite bad.

theandrewbailey · 9 years ago

> These efforts have borne a lot of fruit over the years, but we always run up against the same issue: it's very difficult to make concurrent workloads perform well on CPython.

> To solve this problem, we investigated a number of other Python runtimes. Each had trade-offs and none solved the concurrency problem without introducing other issues.

trotterdylan · 9 years ago

Thanks for trying it out!

Yeah, Grumpy does not currently support old-style classes. Since all of our code internally requires new-style classes, this was not a high priority feature. It is something that we'll get to.

Animats · 9 years ago

That's fascinating. It's creating run-time data structures similar to CPython's for data, and manipulating them with very general code. There seems to be a type comparable to Python's internal CObject, and it's used for most (all?) data. It's not generating Go that looks anything like human-written Go. There's no sign of type inference, although it's hard to tell from such a simple example. It's a lot like a Python run time environment, where everything is a CObject. Still, once you can do that, you can start optimizing, such as inferring that something is an integer and using ordinary Go arithmetic types.

All that stuff with "switch" seems to be to handle Python exceptions in a language that doesn't have exceptions. Maybe later, analysis can tell that some function can't raise an exception, and translated calls for such functions can be simpler.

weberc2 · 9 years ago

Interesting to note that `print()` is supported out of the box--no need to `from __future__ import print_function`.

zardeh · 9 years ago

Pypy still has a GIL. GIL-free seems to be one of the explicit goals of this project.

nemith · 9 years ago

GIL still exists in pypy

alrs · 9 years ago

Because they want to get off of Python completely, and forever. They're not looking for speed, they're looking to ditch the liability of supporting Python and its ecosystem.

Sphax · 9 years ago

It's right there in the post:

> To solve this problem, we investigated a number of other Python runtimes. Each had trade-offs and none solved the concurrency problem without introducing other issues.

ionforce · 9 years ago

What does "hard-code compiler" mean?

poooogles · 9 years ago

>- Amusingly, it runs Python 2.7, even though this project started long after Python 3.x came out.

Python 2.7 is what's running at Google. Not really surprising they're looking at this considering the fast approaching end of (core dev) support for Python 2.7.

Write an interpreter in another language and programatically port modules to Go. Seems pretty sensible to me.

trotterdylan · 9 years ago

> Amusingly, it runs Python 2.7, even though this project started long after Python 3.x came out.

Basically, we needed to support a large existing Python 2.7 codebase. See discussion here: https://github.com/google/grumpy/issues/1

> It's a hard-code compiler, not an interpreter written in Go. That implies some restrictions, but the documentation doesn't say much about what they are. PyPy jumps through hoops to make all of Python's self modification at run-time features work, complicating PyPy enormously. Nobody uses that stuff in production code, and Google apparently dumped it.

There are restrictions. I'll update the README to make note of them. Basically, exec and eval don't work. Since we don't use those in production code at Google, this seemed acceptable.

> If Grumpy doesn't have a Global Interpreter Lock, it must have lower-level locking. Does every built-in data structure have a lock, or does the compiler have enough smarts to figure out what's shared across thread boundaries, or what?

It does fine grained locking. Mutable data structures like lists and dicts do their own locking. Incidentally, this is one reason why supporting C extensions would be complicated.

joshuata · 9 years ago

In regards to the third point: The global interpreter lock protects the fact that python's GC scheme is not thread safe. It does not coordinate accesses across threads, and therefore grumpy's would not either. In grumpy the GIL is replaced by Go's GC implementation that is specifically tuned for multithreaded execution. Any additional synchronization would need to be done with individual locks etc...

TillE · 9 years ago

> Nobody uses that stuff in production code

Nobody uses the features of Python which make it a dynamic language? Google must write some really weird Python if their compiler is that strict.

alrs · 9 years ago

No support for 3 makes sense. This is all about building an off-ramp to put Python behind them.

fiddlerwoaroof · 9 years ago

> - It's a hard-code compiler, not an interpreter written in Go. That implies some restrictions, but the documentation doesn't say much about what they are. PyPy jumps through hoops to make all of Python's self modification at run-time features work, complicating PyPy enormously. Nobody uses that stuff in production code, and Google apparently dumped it.

Does it really imply many restrictions? Common Lisp, for example, is probably more dynamic than Python and it's been a compiled language for ~20 years.

monokrome · 9 years ago

It's written to run Python 2.7 because these problems are largely solved in Python 3, and needs solved for people on Python 2.x versions.

"Upgrade to Python3" is the usual defense to that, but it's not really practical for large companies with software such as YouTube completely written in Python 2.x.

sergiotapia · 9 years ago

I was in second year of college and the Python 2 vs Python 3 was a couple of years running. Is this fight -still- not resolved? I'm not a python developer so I'm out of the loop.

Arkanosis · 9 years ago

Shed Skin is still living at https://shedskin.github.io/ , but hasn't changed much since the last release in 2013.

trotterdylan · 9 years ago

It does not, yet. The standard library is very incomplete at this point.

xigency · 9 years ago

> Until it doesn't, it's not a Python runtime...

True Python runtimes fail the CPython test suite. They have some work to do!

kodablah · 9 years ago

Nice. I've been toying with doing similar for the JVM [0]. I think people are coming to realize that while Go is not a great/powerful language (IMO), the runtime is great. I would not be surprised if more and more people start targetting Go if they want a GC and cross platform static compilation w/out LLVM complications and get a nice stdlib for free.

0 - https://github.com/cretz/goahead

trotterdylan · 9 years ago

YouTube does not run on Grumpy. There is a lot of work left to do before Grumpy can run a large existing codebase.

Twirrim · 9 years ago

Every time I've tried to use Jython, I've found it won't work with pre-existing code all that well.

In part it has exposed CPython "implementation quirks" that people were wittingly or otherwise taking advantage of. In other cases there doesn't seem to be obvious reasons for the differences and has required special-casing the python code to handle it.

It has been great with code written from scratch, specifically for it.

int_19h · 9 years ago

Everyone who likes the numerous new language features, or new syntax for old features, in Python 3?

fletom · 9 years ago

Python 3 is a more modern, better language, that's why. Hopefully, grumpy will support it soon enough, though it's understandable that they started with 2.7 to support Google's huge older codebase.

trotterdylan · 9 years ago

I don't know. I hope so. I think this is an area where Go can succeed and integration with existing Python libraries could be useful.

coldtea · 9 years ago

>One of my complaints about Go is that it seems to be designed with the (mistaken, in my opinion) notion that what we really need is just a better C.

It was designed with the notion that all some people really need is just a better C.

For other people there's Swift, Rust, etc.

weberc2 · 9 years ago

I don't know that it was made with assumption that everyone needs a better C, but certainly that many could benefit from it. I think they were largely correct--I certainly appreciate Go's simplicity and ease of use.

trotterdylan · 9 years ago

> I would love to know how big the codebase is.

Sorry, can't be very specific, but rewriting all the frontend code would take a lot more effort than writing a new Python runtime :)

> It seems like writing a translator to deal with all the use cases is so much more work and risky than iteratively rewriting portions (in whatever faster more concurrent language) and using some form microservice/process message passing to communicate with legacy pieces.

We do iteratively rewrite components as well. We are pursuing multiple strategies.

> Love to know how they compose async operations currently? Is it some sort of object (e.g. Futures, promises, observables, etc)?

Most async operations are performed out-of-process by other servers.

> Is Grumpy going to have some sort of language difference (to Python) to compose async stuff (e.g. async and await)?

I'd love to support async and await at some point.

> Of course being biased towards the JVM (since I know it so well) they could get really fast concurrency if they want with Jython today. Most of the Python tools already work with Jython (assuming 2.7).

We did also do an evaluation of Jython but there were a number of technical issues that made it unsuitable for our codebase and workload. One such example is this longstanding issue: http://bugs.jython.org/issue527524. I just noticed the very recent update on that thread that implemented the workaround outlined in 2010 by Jim Baker. We tried that workaround and found we got a huge performance hit on affected code. There were a few other general performance problems as well but I can't recall all the details.

Please note I'm not at all bashing Jython, I think it's a great project with a sound design, it just wasn't right for us.

> With Jython you could always drop down into Java (or any other JVM lang) if you need more speed as well C for cpython (or even C from Java). It is unclear what you do with Grumpy with performance critical code. Can you interface with Go code or is the plan C?

You can interface with Go code directly, e.g. from the blog post:

  from __go__.net.http import ListenAndServe, RedirectHandler
  handler = RedirectHandler('http://github.com/google/grumpy', 303)
  ListenAndServe('127.0.0.1:8080', handler)

trotterdylan · 9 years ago

Unfortunately the native interface is still pretty immature so I can't guarantee things will work as they should. In this case, I think the problem is that you have a *Response object which is not itself a struct. I've rewritten this part of the code a couple times and I think this functionality was lost. I've filed: https://github.com/google/grumpy/issues/13

Sphax · 9 years ago

How entitled can you be ? Why are they required to do anything of the sort ?

tedmielczarek · 9 years ago

They tried that with CPython, it didn't work: https://www.python.org/dev/peps/pep-3146/

steveklabnik · 9 years ago

The first stop is usually a CONTRIBUTING.md Here's theirs https://github.com/google/grumpy/blob/master/CONTRIBUTING.md

There's not a ton there. The next place I'd go is open issues: https://github.com/google/grumpy/issues

It looks like at the moment they're mostly random bug reports from people who have tried Grumpy since this announcement, rather than ones filed by people working on Grumpy since before it was made public. So that's a bit trickier.

The last place I'd look is then, the README: https://github.com/google/grumpy not a ton there about ways they wish to have people contribute.

At this point, what I'd do personally is open an issue asking how you can get involved; possibly by improving this documentation on how to get involved!

Anyway, that's what I'd do. Hope that helps!

yakubin · 9 years ago

You can probably mail a maintainer or a developer for hints of a project you'd be interested in. Or better you can send a question to the project's mailing list. People there often are very helpful. I've done that. It worked. (I haven't tried contributing to any Google-managed projects though.)

mavhc · 9 years ago

"much" could mean anything, from "what's a PR?" https://guides.github.com/activities/hello-world/

to "what's good etiquette for contributing to open source projects?", I'd say: add as much information as possible to your PR, why it does what it does, how it does it, etc.

webmaven · 9 years ago

Check out OpenHatch: https://openhatch.org/

skybrian · 9 years ago

It's not that different from what's happening with Java (via Android) or Go (with GopherJS, for example), or even C (many, many extensions). A popular language attracts implementors, even if they don't implement the whole thing on their target platform.

viraptor · 9 years ago

Why do you think multiple implementations are a problem? We've got multiple ruby runtimes, multiple basics, multiple JavaScript engines, multiple C compilers, etc. None of those languages are falling apart because of it.

BuckRogers · 9 years ago

GVR, the PSF and the core dev team overestimated their influence. They still truly believe the majority will come around to Python3. I agree with your sentiments and balkanization is the right word. Guido won't even read these comments. He thinks it's all some unjust slander and nonsense that will be forgotten in 3 years. :)

It is a good time to jump off the Python train in general, and I say that as someone invested in Python who loves it. If possible I'd recommend people reach for Go or Elixir depending on their needs or requirements.

I will admit I'm a little shocked how much of a failure Python3 adoption has been. I think if it had been Grumpy from the start it would've been a huge success. This is exactly what people want and Google should be commended for sharing this.

Here's to hoping Grumpy takes on a life of its own and is the new de facto Python.

hellofunk · 9 years ago

Actually, historical evidence suggests the opposite of your argument: multiple implementations of a language typically mean it is succeeding, not failing.

porker · 9 years ago

> His stubborn view on functional mechanisms

Reference, for a non-Python dev who hasn't kept up with it?

vorg · 9 years ago

Maybe you don't like Grumpy as a name. I think it was named after NumPy (pr. ˈnʌmpaɪ) so I guess it's pronounced ˈgrʌmpaɪ rather than ˈgrʌmpi:

Around 2011 the beta releases of the static typing additions to Apache Groovy were called "grumpy" but the name was dropped after objections from the Grails crowd. I think the quality of the product is more important than the name, so Grumpy should do OK regardless if it's built and maintained properly.

webmaven · 9 years ago

> Could they not just use multiprocessing instead of threading for the cpu-heavy parts of the YouTube codebase, and threading or asyncio for the io-bound parts of it?

Probably, but consider how much bare iron Google has lying around, much of it likely with few cores-per-CPU. Using it efficiently and not accelerating it's obsolescence is probably a priority for them.

BTW, does anyone have data that would suggest how long it does take an org at the scale of an Amazon, Google, or Facebook to entirely replace their HW? I assume that it isn't only through attrition, and that Google for example currently has no servers running that date back to Y2k, but I have no idea what the "half-life" of a server is at their scale.