Comments
yourfanat wrote: I am using another tool for Oracle developers - dbForge Studio for Oracle. This IDE has lots of usefull features, among them: oracle designer, code competion and formatter, query builder, debugger, profiler, erxport/import, reports and many others. The latest version supports Oracle 12C. More information here.
Cloud Expo on Google News
SYS-CON.TV
Cloud Expo & Virtualization 2009 East
PLATINUM SPONSORS:
IBM
Smarter Business Solutions Through Dynamic Infrastructure
IBM
Smarter Insights: How the CIO Becomes a Hero Again
Microsoft
Windows Azure
GOLD SPONSORS:
Appsense
Why VDI?
CA
Maximizing the Business Value of Virtualization in Enterprise and Cloud Computing Environments
ExactTarget
Messaging in the Cloud - Email, SMS and Voice
Freedom OSS
Stairway to the Cloud
Sun
Sun's Incubation Platform: Helping Startups Serve the Enterprise
POWER PANELS:
Cloud Computing & Enterprise IT: Cost & Operational Benefits
How and Why is a Flexible IT Infrastructure the Key To the Future?
Click For 2008 West
Event Webcasts
Tracing Celery Performance for Web Applications
A few useful ways to get insight into Celery performance when running tasks.

Are you using Celery to process Python back-end tasks asynchronously?  Have you wanted to get insight into their resource consumption and efficiency?  Here’s a few useful ways to get insight into Celery performance when running tasks.

A simple celery task
For a quick review, Celery lets you turn any python method into an asynchronous task.  Here’s a simple one:

1
2
3
4
from celery.task import task
@task
def add(x, y):
return x + y

Let’s trace Celery
We’ll start with the good stuff. In the latest release of our Python instrumentation, oboeware-1.0, we have an updated API that makes it super-easy to gather performance data from any Python code, including Celery tasks. Here’s how we’d add tracing to our example task:

Celery-Oboe

The key part is the @oboe.Context.trace decorator.  Note that we’re assigning to the keys Controller and Action.  This will be used by Tracelytics to segment the data.  You can also optionally use the keys HTTP-Host and URL to indicate domain and URL to Tracelytics.  Here’s what the data we’re gathering looks like in the dashboard (I added a few sqlalchemy queries to spice up the data):

celery performance in tracelytics

The code at the top is configuration tunables: set the sample rate to a fractional value to trace only a fraction of your Celery tasks.

Event-based workers
You might be using event-based workers to save a few bytes of ram.  Tracelytics also supports eventlet-based workers.  However, you’ll need to install our Tracelytics-enabled greenlet module using pip or easy install:

1
$ pip install --extra-index-url=http://pypi.tracelytics.com greenlet-0.3.4-tly1.0

Slightly more interesting
Of course, you’re probably interested in more complex celery tasks than that.  The good news is, all of our normal instrumentation works with your celery workers.  Here’s a screenshot of one of our internal Celery workers in action:

tracing complex celery tasks

Other resources
Looking for different tools?  Good news–there’s a pretty healthy Celery ecosystem.  Here’s a few useful packages and I’m sure there’s tons I’m forgetting–leave comments for anything else you find helpful with Celery!

celery performance

  • If you want to keep track of worker status, you might be interested in celerymon, a monitor for celery tasks.  It keeps tabs of task execution and workers.
  • Trying to track down a memory leak?  Consider running dowser inside your celery worker.

Photo Credit: dottiemae

Related Articles

Tracing Python — An API

Python and gevent

Advanced SQLAlchemy & Performance Concerns

About Dan Kuebrich
Dan Kuebrich is a web performance geek, currently working on Application Performance Management at AppNeta. He was previously a founder of Tracelytics (acquired by AppNeta), and before that worked on AmieStreet/Songza.com.

Latest Cloud Developer Stories
Your job is mostly boring. Many of the IT operations tasks you perform on a day-to-day basis are repetitive and dull. Utilizing automation can improve your work life, automating away the drudgery and embracing the passion for technology that got you started in the first place. In...
The challenges of aggregating data from consumer-oriented devices, such as wearable technologies and smart thermostats, are fairly well-understood. However, there are a new set of challenges for IoT devices that generate megabytes or gigabytes of data per second. Certainly, the i...
Whenever a new technology hits the high points of hype, everyone starts talking about it like it will solve all their business problems. Blockchain is one of those technologies. According to Gartner's latest report on the hype cycle of emerging technologies, blockchain has just ...
CloudEXPO New York 2018, colocated with DevOpsSUMMIT and DXWorldEXPO New York 2018 will be held November 12-13, 2018, in New York City and will bring together Cloud Computing, FinTech and Blockchain, Digital Transformation, Big Data, Internet of Things, DevOps, AI and Machine Le...
CloudEXPO | DevOpsSUMMIT | DXWorldEXPO are the world's most influential, independent events where Cloud Computing was coined and where technology buyers and vendors meet to experience and discuss the big picture of Digital Transformation and all of the strategies, tactics, and to...
Subscribe to the World's Most Powerful Newsletters
Subscribe to Our Rss Feeds & Get Your SYS-CON News Live!
Click to Add our RSS Feeds to the Service of Your Choice:
Google Reader or Homepage Add to My Yahoo! Subscribe with Bloglines Subscribe in NewsGator Online
myFeedster Add to My AOL Subscribe in Rojo Add 'Hugg' to Newsburst from CNET News.com Kinja Digest View Additional SYS-CON Feeds
Publish Your Article! Please send it to editorial(at)sys-con.com!

Advertise on this site! Contact advertising(at)sys-con.com! 201 802-3021



SYS-CON Featured Whitepapers
ADS BY GOOGLE