May 13, 2014
TechnologyFinal Effect
Following is the final effect of our own app:
There are lots to be finalize and optimized, but currently It could be ful-fill our requirements which retriving the data and generate the flot.
The next series I will try to write some ruby or node.js programs which did the same functionalities, to compare the differencies between app developement.
Also to write a web-proxy is a work full of challenge, this will be took as next consideration of developing apps on heroku .
May 12, 2014
TechnologyUnderstanding the flask and Jinja
Flask Example
hello1.py is listed as following:
from flask import Flask
app = Flask(__name__)
@app.route("/")
def index():
return 'Index Page'
@app.route('/hello')
def hello():
return "Hello World!"
@app.route('/hello1')
def hello1():
return "Hello World 1!"
if __name__ == "__main__":
app.run()
Run this via:
$ python hello1.py
Then use your browser for visiting http://localhost:5000, http://localhost:5000/hello, http://localhost:5000/hello1. You will view different output result.
Jinja Example
The sample.py is listed as following:
# Load the jinja library's namespace into the current module.
import jinja2
# In this case, we will load templates off the filesystem.
# This means we must construct a FileSystemLoader object.
#
# The search path can be used to make finding templates by
# relative paths much easier. In this case, we are using
# absolute paths and thus set it to the filesystem root.
templateLoader = jinja2.FileSystemLoader( searchpath="/" )
# An environment provides the data necessary to read and
# parse our templates. We pass in the loader object here.
templateEnv = jinja2.Environment( loader=templateLoader )
# This constant string specifies the template file we will use.
TEMPLATE_FILE = "//home/Trusty/code/python/heroku/Jinja2/example1.jinja"
# Read the template file using the environment object.
# This also constructs our Template object.
template = templateEnv.get_template( TEMPLATE_FILE )
# Specify any input variables to the template as a dictionary.
templateVars = { "title" : "Test Example",
"description" : "A simple inquiry of function." }
# Finally, process the template to produce our final text.
outputText = template.render( templateVars )
print outputText
Create the example1.jinja at the corresponding directory, contains following content:
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<title>{{ title }}</title>
<meta name="description" content="{{ description }}" />
</head>
<body>
<div id="content">
<p>Why, hello there!</p>
</div>
</body>
</html>
Then the result will viewed as following:
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<title>Test Example</title>
<meta name="description" content="A simple inquiry of function." />
</head>
<body>
<div id="content">
<p>Why, hello there!</p>
</div>
</body>
</html>
Rendering Template Returning
First create the template file under the directory ‘templates’, this is the default position for flask for searching the template files:
$ mkdir templates
$ cat layout.html
<style type="text/css">
.metanav
{
background-color: yellow;
}
</style>
<div class="page">
<h1>Flaskr</h1>
<div class="metanav">
{{ a_random_string }}
{{ a_random_list[3] }}
</div>
</div>
Then in the genhtml.py, we add the following lines for let the template system to rendering our pages:
from flask import render_template
@app.route('/index')
# Generate the index page, for debugging now
def index():
# Use template for rendering the content
rand_list= [0, 1, 2, 3, 4, 5]
return render_template('layout.html', a_random_string="Heey, what's up!", a_random_list=rand_list)
Now browser you http://localhost:5000/index, you can see the template rendered result.
May 11, 2014
TechnologyWe will continue to deploy our tasks on heroku. In this article we will finish the data retriving and database insertion.
Honcho
Install and Configuration of Honcho:
$ pip install honcho
# Regenerate requirement.txt and upload it onto heroku
$ mv Procfile ProcfileHoncho
# Edit the new Procfile:
$ vim Procfile
web: honcho -f ProcfileHoncho start
In local, we can also use following command for swiftly verifying our code:
# Because our user belongs to root group, so set following variable firstly
$ export C_FORCE_ROOT=1
$ foreman start
Now you can visit http://localhost:5000 for viewing the result.
task.py
Following is the script for tasks.py:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
# Set default encoding: utf-8
import sys;
reload(sys);
# Setting the utf-8 format
sys.setdefaultencoding("utf8")
import logging
import string
from celery import Celery
from celery.task import periodic_task
from datetime import datetime,timedelta
from os import environ
# For retrieving temperature/humidity
import pywapi
import urllib2
from urllib2 import ProxyHandler
import re
from BeautifulSoup import BeautifulSoup
# For using database(Postgresql)
# from flash.ext.sqlalchemy import SQLAlchemy
from sqlalchemy import create_engine
from sqlalchemy import MetaData, Column, Table, ForeignKey
from sqlalchemy import Integer, String, DateTime
# redis configuration for celery
### Local, should be change to remote when deploying to heroku
REDIS_URL = environ.get('REDISTOGO_URL', 'redis://localhost')
celery = Celery('tasks', broker=REDIS_URL)
# Fetch
# Define Periodic Tasks
# @periodic_task(run_every=timedelta(minutes=60))
# Funciton for fetching data and store it in Postgres
def fetch_and_store_data():
# Fetching the weather information from Yahoo.
Yahoo_Result = pywapi.get_weather_from_yahoo('CHXX0099')
Current_Temp = string.lower(Yahoo_Result['condition']['temp'])
Current_Humi = string.lower(Yahoo_Result['atmosphere']['humidity'])
Tomorrow_Forecast = Yahoo_Result['forecasts'][0]
Twenty_Four_Hours = Yahoo_Result['forecasts'][1]
Fourty_Eight_Hours = Yahoo_Result['forecasts'][2]
Seventy_Two_Hours = Yahoo_Result['forecasts'][3]
# !!! comment proxy related for deploying to heroku !!! #
proxy = urllib2.ProxyHandler({'http': '192.11.236.225:8000'})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)
page = urllib2.urlopen("http://www.pm25.in/nanjing")
soup = BeautifulSoup(page) #
# Find the detailed table from the soup. #
table = soup.find('table', {'id':'detail-data'})#
# Fetch the XuanWuHu, if not, use MaiGaoQiao ins#tead.
rows = table.findAll('tr') #
for subrows in rows: #
if "玄武湖" in subrows.text: #
XuanwuLake = subrows #
else: #
if "迈皋桥" in subrows.text: #
XuanwuLake = subrows #
XuanwuLake_subitem = XuanwuLake.findAll('td') #
# Here we will get an array, fetch out the text #for the content from this array.
# Fetched origin data, different from cnpm25.cn #
pm_25_orig = XuanwuLake_subitem[4].text #
pm_10_orig = XuanwuLake_subitem[5].text
# Open the Database
engine = create_engine('postgresql+psycopg2://Trusty:@localhost:5432/mylocaldb',echo=True)
metadata=MetaData(bind=engine)
# Definition of the weather table
weather_table=Table('weather',metadata,
Column('Insert_Time', DateTime(timezone=True),primary_key=True),
Column('Temperature', Integer),
Column('Humidity', Integer),
Column('PmTen', Integer),
Column('PmTwoFive', Integer),
)
# Create the table in mylocaldb
metadata.create_all(checkfirst=True)
# Record Insertion
# First generate an insertion sentense:
ins = weather_table.insert()
# Really insert into the Database
# ins = weather_table.insert().values(Insert_Time=datetime.datetime.utcnow(), Temperature=25, Humidity=75, PmTen=100, PmTwoFive=55) # Example
ins = weather_table.insert().values(Insert_Time=datetime.utcnow(), Temperature=Current_Temp, Humidity=Current_Humi, PmTen=int(pm_10_orig), PmTwoFive=int(pm_25_orig))
# Connect to engine and execute.
conn = engine.connect()
result = conn.execute(ins)
#return Current_Temp
return pm_25_orig
@periodic_task(run_every=timedelta(seconds=10))
def print_fib():
#logging.info(fib(30))
logging.info("This could be viewed in logging!")
Notice, this version only works in local.
For updating the real database on heroku, we have to do some modification on redis server and remove the proxy server, these are the works we need to done in next chapter.
May 11, 2014
TechnologyDeploy on Heroku
The deployed version is listed as following:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
# Set default encoding: utf-8
import sys;
reload(sys);
# Setting the utf-8 format
sys.setdefaultencoding("utf8")
import os
import logging
import string
from celery import Celery
from celery.task import periodic_task
from datetime import datetime,timedelta
from os import environ
# For retrieving temperature/humidity
import pywapi
import urllib2
from urllib2 import ProxyHandler
import re
from BeautifulSoup import BeautifulSoup
# For using database(Postgresql)
# from flash.ext.sqlalchemy import SQLAlchemy
from sqlalchemy import create_engine
from sqlalchemy import MetaData, Column, Table, ForeignKey
from sqlalchemy import Integer, String, DateTime
# redis configuration for celery
### Local, should be change to remote when deploying to heroku
# REDIS_URL = environ.get('REDISTOGO_URL', 'redis://localhost')
### Heroku Ways
REDIS_URL = environ.get('REDISCLOUD_URL')
celery = Celery('tasks', broker=REDIS_URL)
# Fetch
# Define Periodic Tasks
@periodic_task(run_every=timedelta(minutes=60))
# Funciton for fetching data and store it in Postgres
def fetch_and_store_data():
# Fetching the weather information from Yahoo.
Yahoo_Result = pywapi.get_weather_from_yahoo('CHXX0099')
Current_Temp = string.lower(Yahoo_Result['condition']['temp'])
Current_Humi = string.lower(Yahoo_Result['atmosphere']['humidity'])
Tomorrow_Forecast = Yahoo_Result['forecasts'][0]
Twenty_Four_Hours = Yahoo_Result['forecasts'][1]
Fourty_Eight_Hours = Yahoo_Result['forecasts'][2]
Seventy_Two_Hours = Yahoo_Result['forecasts'][3]
# !!! comment proxy related for deploying to heroku !!! #
# proxy = urllib2.ProxyHandler({'http': '1xx.x.xx.xxx:2xxx'})
# opener = urllib2.build_opener(proxy)
# urllib2.install_opener(opener)
# !!! comment out end #
page = urllib2.urlopen("http://www.pm25.in/nanjing")
soup = BeautifulSoup(page) #
# Find the detailed table from the soup. #
table = soup.find('table', {'id':'detail-data'})#
# Fetch the XuanWuHu, if not, use MaiGaoQiao ins#tead.
rows = table.findAll('tr') #
for subrows in rows: #
if "玄武湖" in subrows.text: #
XuanwuLake = subrows #
else: #
if "迈皋桥" in subrows.text: #
XuanwuLake = subrows #
XuanwuLake_subitem = XuanwuLake.findAll('td') #
# Here we will get an array, fetch out the text #for the content from this array.
# Fetched origin data, different from cnpm25.cn #
pm_25_orig = XuanwuLake_subitem[4].text #
pm_10_orig = XuanwuLake_subitem[5].text
# Open the Database
# Local Engine
# engine = create_engine('postgresql+psycopg2://Trusty:@localhost:5432/mylocaldb',echo=True)
# Heroku Engine
db_conn = environ['DATABASE_URL']
engine = create_engine(db_conn)
metadata=MetaData(bind=engine)
# Definition of the weather table
weather_table=Table('weather',metadata,
Column('Insert_Time', DateTime(timezone=True),primary_key=True),
Column('Temperature', Integer),
Column('Humidity', Integer),
Column('PmTen', Integer),
Column('PmTwoFive', Integer),
)
# Create the table in mylocaldb
metadata.create_all(checkfirst=True)
# Record Insertion
# First generate an insertion sentense:
ins = weather_table.insert()
# Really insert into the Database
# ins = weather_table.insert().values(Insert_Time=datetime.datetime.utcnow(), Temperature=25, Humidity=75, PmTen=100, PmTwoFive=55) # Example
ins = weather_table.insert().values(Insert_Time=datetime.utcnow(), Temperature=Current_Temp, Humidity=Current_Humi, PmTen=int(pm_10_orig), PmTwoFive=int(pm_25_orig))
# Connect to engine and execute.
conn = engine.connect()
result = conn.execute(ins)
#return Current_Temp
return pm_25_orig
# Every 3 minutes we will see debug information, from heroku log
# @periodic_task(run_every=timedelta(minutes=3))
# @periodic_task(run_every=timedelta(seconds=3))
# def print_fib():
# #logging.info(fib(30))
# logging.info("This could be viewed in logging!")
This version use remote service, and it will handle the fetching/updating every 60 minutes.
Prevent Heroku from Sleeping
Register a service at https://uptimerobot.com/, it will automatically ping or get http(s) service from your web apps.
Until now, almost all of the back-end has been set up. We can fetch the data and periodly insert into the database, and our web app will continue to run(Never sleep).
Tomorrow we will try to write a beautiful front-end for our webapp.
May 10, 2014
TechnologyInstallation
Install and run rabbitmq server via:
$ yaourt rabbitmq
$ rabbitmq-server
Install celery in python virtual enviroment:
$ workon venv2
(venv2) $ pip install celery
Run Simple Tasks
Following python file, named “tasks.py” defines two tasks:
from celery import Celery
app = Celery('tasks', backend='amqp', broker='amqp://')
@app.task(ignore_result=True)
def print_hello():
print 'hello there'
@app.task
def gen_prime(x):
multiples = []
results = []
for i in xrange(2, x+1):
if i not in multiples:
results.append(i)
for j in xrange(i*i, x+1, i):
multiples.append(j)
return results
Run it via:
celery worker -A tasks &
Now in another terminal you can use python interractive window for:
from tasks import print_hello
from tasks import gen_prime
print_hello()
primes = gen_prime(1000)
primes = gen_prime(50000)
# CTRL+C will stop it.
# Access the background worker
primes = gen_prime.delay(50000)
# by the worker executing now, because we configured the backend for application
primes.ready()
False
...
True
print primes.get()
Periodic Tasks
Add following lines into the tasks.py:
from celery.task import periodic_task
from datetime import timedelta
@periodic_task(run_every=timedelta(minutes=1))
def print_minutes():
print 'Hello, 1 minute reached'
@periodic_task(run_every=timedelta(seconds=3))
def every_3_seconds():
print("Running periodic task!")
Now start the tasks.py via following commands:
$ celery -A tasks worker --loglevel=info --beat
You will see Celery output “Running periodic tasks!” every 3 seconds, while every 1 minutes the “Hello, 1 minute reached” will be printed out.
To Be Thought
How to mirgrate it with our heroku web app ?
The periodic_task is good for fetching pages/datas and generate the result, then store the results into the database.
But the webapp should response to user’s http request, this will be the main task.
