Write Python Weather APP on Heroku(4)

Local Database Sync

First fetch the remote database to localdb via following command:

$ heroku pg:pull DATABASE_URL mylocaldb --app  python-weather-app

This command will pull down your database down and create a copy version locally. You can easily view all of the database via psql mylocaldb.

As root, edit following files:

# pwd
/var/lib/postgres/data
# vim postgresql.conf
listen_addresses = 'localhost'		# what IP address(es) to listen on;
					# comma-separated list of addresses;
					# defaults to 'localhost'; use '*' for all
					# (change requires restart)
port = 5432				# (change requires restart)
# vim pb_hba.conf
# IPv4 local connections:
host    all             all             127.0.0.1/32            trust
host    all             all             127.0.0.1/32            md5

After editing, save the file and restart the postgresql service.

Connecting to LocalDatabase

Following shows how to add/query the records. Now your environment could be totally debugged locally.

>>> import psycopg2
>>> psycopg2.connect(database="mylocaldb",user="Trusty",password="",host="127.0.0.1",port="5432")
<connection object at 0x30f0cd0; dsn: 'dbname=mylocaldb user=Trusty password=xx host=127.0.0.1 port=5432', closed: 0>
>>> db_conn = 'postgresql+psycopg2://Trusty:@localhost/mylocaldb'
>>> app = Flask(__name__) 
>>> app.config['SQLALCHEMY_DATABASE_URI'] = db_conn
>>> db = SQLAlchemy(app)
>>> db.create_all()
>>> class User(db.Model):
...     id = db.Column(db.Integer, primary_key=True)
...     name = db.Column(db.String(80))
...     email = db.Column(db.String(120), unique=True)
...     def __init__(self, name, email):
...         self.name = name
...         self.email = email
...     def __repr__(self):
...         return '<Name %r>' % self.name
... 
>>> user = User('John Doe', 'john.doe@example.com')
>>> all_users=User.query.all()
>>> print all_users
[<Name u'John Doe'>]
>>> user1 = User('Jim Green', 'jim.green@english.com')
>>> db.session.add(user1)
>>> db.session.commit()

In genhtml.py we need to do corresponding changes in order to enable the local database.

Use Tunnel For Acrossing Something

Network Envorinment Introduction

The network envoriment in daily working envoriment is very bad, thus I have to think for a solution, which could improve my network speed.
Following picture describes the network topology of the daily working.

From the picture we can see, several users shared a very narrow path, and this path have to go through chinese firewall, this firewall is ghastly, because it will filter some sensitive website which is not welcomed by CN gov.

Our VPN Introduction

There are very wide VPN(Virtual Private Network) between CN and US, the US networking don’t have to pass through the firewall.
Another big surprise is created by the time difference, when chinese are working, lots of american people are out of office.

Surely we can make full use of our whole company’s network condition.

Solution 1: SSH Forwarding

First we will find a server which could forward ssh, just like in the picture.

Then we can use following command for setup a ssh tunnel, which could forwarding our network flow to us proxy:

$ ssh -C  -L YourMachine:Port:USProxy:USProxy_Port YouAccount@ForwardingServer

Then in your browser or your application, use http://YourMachine:Port as a proxy.

Solution 2: TCP Tunnel Forwarding

Not every server can open ssh forwarding for you. For example, in following server, tcp forwarding is forbidden:

$ cat /etc/ssh/sshd_config
# Port forwarding
AllowTcpForwarding no

Thus we have to setup our own tcp tunnel manually.

Netcat Way

Use following way you can use netcat for creating a very simple tunnel, which could forwarding all of your flow to US proxy, these operation have to be done on server:

$ mkdir /tmp/fifo
$ nc -lvvp -k 2323 0</tmp/fifo | nc -k USproxy USProxy_Port 1>fifo

Then set the local proxy to http://server_ip:2323, then you can reached the proxy.

Tunnel Way

Netcat way is OK, but the netcat version is very old on US Server, it can’t support ‘-k’ option, for ‘-k’ option is only supported by openbsd-netcat, and because the server is too old(It’s Sun OS 5.10, or solaris? ), so we have to find other ways.
Luckily I find a small tool, which could fit for our requirement.

$ wget http://www.cri.ensmp.fr/~coelho/tunnel.c
$ gcc -o tunnel tunnel.c -lsocket

This compiling will complain ‘herror’ is not supported, thus we have to comment them, or change them from ‘herror’ to ‘printf’, anyway, the error happens seldomly.
Use following command for setting up a tunnel in server:

$ ./tunnel -Lr server_ip:1080 proxy:80

Then in your own PC, set proxy to http://server_ip:1080, you can reach the internet through your own tunnel, which will guide your traffic from VPN to US, then to Internet.

Make Tunnel Invisible

Normally system administrator won’t like tunnel on server, maybe they will scan the server and find out the port occupation. So we have to do some modification to tunnel.c.
First, change the name of the executable file:

$ mv tunnel.c autrace.c
$ gcc -o autrace autrace.c -lsocket

So now, you can run your tunnel program via ‘./autrace -Lr localhost:1080 proxy:80’.

But this is not so safe, administrator will also find the port, then they will track this port, and find your tricks, so we have to hidden the port words.
In autrace.c, do following changes in corresponding lines:

//  Around line 128, change the ip/port into your own. 

/* default connexion. */
#define LHOST "138.138.138.138" /* this really means 127.0.0.1, thus no network! */
#define LPORT "4444"
/* DHOST: <same as chosen LHOST> */
//#define DPORT "23"        /* telnet port */
#define DHOST "139.139.139.139"
#define DPORT "8888"

// Around line 1023, this is actually a bug.  
 dhosts = getenv("DHOST"); 1023

Now you can run command like:

$ ./autracce -s

Make Tunnel Only Serve for you

We have to forbidden other user use our tunnel, because http://server_ip:port is open to all of the person in VPN.
We add following ACL rule in the autrace.c:

// in main(), around line 935
  /* Initialize the parameter for ACL(Access Control List) */
  struct sockaddr_in sa;
  inet_pton(AF_INET, "Your_IP_Address", &(sa.sin_addr));
  allow_address = ntohl(sa.sin_addr.s_addr);

// in main(), around line 1187
      /* In here, we will do filter, filter specified ip address */
      /* Compare the allowed ip address with the incomming's ip address */
      if( allow_address != (ntohl(client_addr.sin_addr.s_addr)))
      {
        fprintf(stderr, "Sorry, you are not welcomed!\n");
        /* No more receive/send any data */
        shutdown(client_socket, 2);
      }

The code will check the incoming client’s ip address, and comparing it to our pre-defined ip address(Your_IP_Address), if they are not equal, our server will directly close the socket, so the client will receive refuse information.

Now you have a very safe and reliable path will will let you reach the internet via wide VPN and swift US network, enjoy it.

Wake On LAN

See if your equipment support “Wake On LAN” feature:

$ ethtool enp0s25 | grep "Wake"
Cannot get wake-on-lan settings: Operation not permitted

If you got this feature, then install wol:

$ pacman -S wol

Record the mac address of your equipment which you want to wake up, in a living machine, if you want to wake it, simply use following command:

# wol -i HOSTNAME_OR_IP MACADDRESS

The next consideration is, how to keep a wake-up equipment 24-hours, I suggest you use BeagleBone or Raspberry PI, or you can research how to use arduino and write your own applications.

Download Android Source Code on RaspberryPI

Just a try. I don’t think I will use raspberryPI for developing, but using it for downloading code is a good idea.
###Go Back Home My raspberryPI is behind the router, so I have to use a ssh tunnel to reach raspberryPI.
Setting up tunnel:

$ ssh -L 2230:10.0.0.230:22 Tomcat.xxx.xx.xxx -l root

Login on local port:

$ ssh root@localhost -p 2230

Now we have a terminal which could reach raspberry PI.
###Package Preparation Since the OS on my raspberryPI is ArchLinux, I have to install following packages:

$ pacman -S w3m tmux lynx

###Account Setting Use w3m for accessing https://android.googlesource.com/new-password

But, remember, I have a BBB which also runs at home, so I can also use it.

$ apt-get install elinks

Use elinks for making connection to https://android.googlesource.com/new-password, remember the username, and the machine and login name, just copy them into your ~/.netrc, and make sure you have the right priviledge for the file ~/.netrc.
###Repo sync This will take for a long~long time, depending on your bandwidth:

repo init -u https://android.googlesource.com/a/platform/manifest -b master

then we use the following file for sync the repo:

#!/bin/bash

while [ 1 ]
do
    repo sync -j8
    if [ "$?" = "0" ] ; then
        echo "rsync completed normally"
        exit
    else
        echo "Rsync failure. Backing off and retrying..."
        sleep 1
    fi
done

Now call tmux for holding the sync procedure:

$ tmux new -s Android
$ /bin/bash ./autodown.sh 2>&1 | tee autodown.log
$ ctrl+b d

The sync process now is held in tmux session. Let it go, next time when you want to see the procedure, just ssh to the board use:

tmux a -t Android

Deploy Snova On Heroku

###Setting Account First you have to install heroku deploy tool via:

$ heroku plugins:install https://github.com/heroku/heroku-deploy 

Then login and create the app:

$ heroku login
Enter your Heroku credentials.
Email: xxx@gmail.com
Password (typing will be hidden): 
Authentication failed.
$ $ heroku apps:create xxx
Creating xxx... done, stack is cedar
http://xxx.herokuapp.com/ | git@heroku.com:xxx.git

###Source Code Preparation Download the source code from https://code.google.com/p/snova/, here you need to cross the Greatwall.

wget https://code.google.com/p/snova/downloads/detail?name=snova-c4-java-server-0.22.0.war
wget https://code.google.com/p/snova/downloads/detail?name=snova-0.22.0.zip&can=2&q=

Here, we use the java version of the snova.
###Deploy Apps On Heroku Now begin to deploy your Snova:

heroku deploy:war --war ./snova-c4-java-server-0.22.0.war --app "Your_App_Name" 

###Configure Client unzip the snova-0.22.0.zip and configure it:

unzip snova-0.22.0.zip
cd snova-0.22.0
vim conf/snova.conf
[C4]
Enable=1
WorkerNode[0]=Your_App_Name.herokuapp.com 

You can create a shortcut for calling the Snova, add following line into ~/.bashrc:

alias snova='/home/Trusty/code/gfw/snova/snova-0.22.0/bin/start.sh'

###Configure Browser The proxy address is 127.0.0.1, the port is 48100 or 48102.