Comments
yourfanat wrote: I am using another tool for Oracle developers - dbForge Studio for Oracle. This IDE has lots of usefull features, among them: oracle designer, code competion and formatter, query builder, debugger, profiler, erxport/import, reports and many others. The latest version supports Oracle 12C. More information here.
Cloud Expo on Google News
SYS-CON.TV
Cloud Expo & Virtualization 2009 East
PLATINUM SPONSORS:
IBM
Smarter Business Solutions Through Dynamic Infrastructure
IBM
Smarter Insights: How the CIO Becomes a Hero Again
Microsoft
Windows Azure
GOLD SPONSORS:
Appsense
Why VDI?
CA
Maximizing the Business Value of Virtualization in Enterprise and Cloud Computing Environments
ExactTarget
Messaging in the Cloud - Email, SMS and Voice
Freedom OSS
Stairway to the Cloud
Sun
Sun's Incubation Platform: Helping Startups Serve the Enterprise
POWER PANELS:
Cloud Computing & Enterprise IT: Cost & Operational Benefits
How and Why is a Flexible IT Infrastructure the Key To the Future?
Click For 2008 West
Event Webcasts
Tracing Celery Performance for Web Applications
A few useful ways to get insight into Celery performance when running tasks.

Are you using Celery to process Python back-end tasks asynchronously?  Have you wanted to get insight into their resource consumption and efficiency?  Here’s a few useful ways to get insight into Celery performance when running tasks.

A simple celery task
For a quick review, Celery lets you turn any python method into an asynchronous task.  Here’s a simple one:

1
2
3
4
from celery.task import task
@task
def add(x, y):
return x + y

Let’s trace Celery
We’ll start with the good stuff. In the latest release of our Python instrumentation, oboeware-1.0, we have an updated API that makes it super-easy to gather performance data from any Python code, including Celery tasks. Here’s how we’d add tracing to our example task:

Celery-Oboe

The key part is the @oboe.Context.trace decorator.  Note that we’re assigning to the keys Controller and Action.  This will be used by Tracelytics to segment the data.  You can also optionally use the keys HTTP-Host and URL to indicate domain and URL to Tracelytics.  Here’s what the data we’re gathering looks like in the dashboard (I added a few sqlalchemy queries to spice up the data):

celery performance in tracelytics

The code at the top is configuration tunables: set the sample rate to a fractional value to trace only a fraction of your Celery tasks.

Event-based workers
You might be using event-based workers to save a few bytes of ram.  Tracelytics also supports eventlet-based workers.  However, you’ll need to install our Tracelytics-enabled greenlet module using pip or easy install:

1
$ pip install --extra-index-url=http://pypi.tracelytics.com greenlet-0.3.4-tly1.0

Slightly more interesting
Of course, you’re probably interested in more complex celery tasks than that.  The good news is, all of our normal instrumentation works with your celery workers.  Here’s a screenshot of one of our internal Celery workers in action:

tracing complex celery tasks

Other resources
Looking for different tools?  Good news–there’s a pretty healthy Celery ecosystem.  Here’s a few useful packages and I’m sure there’s tons I’m forgetting–leave comments for anything else you find helpful with Celery!

celery performance

  • If you want to keep track of worker status, you might be interested in celerymon, a monitor for celery tasks.  It keeps tabs of task execution and workers.
  • Trying to track down a memory leak?  Consider running dowser inside your celery worker.

Photo Credit: dottiemae

Related Articles

Tracing Python — An API

Python and gevent

Advanced SQLAlchemy & Performance Concerns

About Dan Kuebrich
Dan Kuebrich is a web performance geek, currently working on Application Performance Management at AppNeta. He was previously a founder of Tracelytics (acquired by AppNeta), and before that worked on AmieStreet/Songza.com.

Latest Cloud Developer Stories
DX World EXPO, LLC, a Lighthouse Point, Florida-based startup trade show producer and the creator of "DXWorldEXPO® - Digital Transformation Conference & Expo" has announced its executive management team. The team is headed by Levent Selamoglu, who has been named CEO. "Now is the ...
"Space Monkey by Vivent Smart Home is a product that is a distributed cloud-based edge storage network. Vivent Smart Home, our parent company, is a smart home provider that places a lot of hard drives across homes in North America," explained JT Olds, Director of Engineering, an...
SYS-CON Events announced today that Conference Guru has been named “Media Sponsor” of the 22nd International Cloud Expo, which will take place on June 5-7, 2018, at the Javits Center in New York, NY. A valuable conference experience generates new contacts, sales leads, potentia...
DevOps is under attack because developers don’t want to mess with infrastructure. They will happily own their code into production, but want to use platforms instead of raw automation. That’s changing the landscape that we understand as DevOps with both architecture concepts (Clo...
In his Opening Keynote at 21st Cloud Expo, John Considine, General Manager of IBM Cloud Infrastructure, led attendees through the exciting evolution of the cloud. He looked at this major disruption from the perspective of technology, business models, and what this means for enter...
Subscribe to the World's Most Powerful Newsletters
Subscribe to Our Rss Feeds & Get Your SYS-CON News Live!
Click to Add our RSS Feeds to the Service of Your Choice:
Google Reader or Homepage Add to My Yahoo! Subscribe with Bloglines Subscribe in NewsGator Online
myFeedster Add to My AOL Subscribe in Rojo Add 'Hugg' to Newsburst from CNET News.com Kinja Digest View Additional SYS-CON Feeds
Publish Your Article! Please send it to editorial(at)sys-con.com!

Advertise on this site! Contact advertising(at)sys-con.com! 201 802-3021



SYS-CON Featured Whitepapers
ADS BY GOOGLE