1. 背景说明

谁提出的需求?什么情况下提出的?

2. 目标客户

产品的使用者是谁?

3. 解决的问题

目标客户为什么要用这个产品?

4. 风险预测

法律法规

5. 功能总览

核心功能有哪些?能干什么?

6. 功能详情

所有的前台功能和后台功能,以及功能的业务规则、界面原型、执行者、前置条件、后置条件。

7. 上、下线要求

想要什么时候上线?满足什么条件时下线?

8. 运营计划

如何获取用户?如何留住用户?

9. 附录

《其他文档》

关键词:debian、etcd 时间:2015年

root@debian:~#
root@debian:~# lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 8.2 (jessie)
Release:	8.2
Codename:	jessie
root@debian:~#
root@debian:~# wget http://ftp.cn.debian.org/debian/pool/main/e/etcd/etcd_2.2.0+dfsg-2_amd64.deb
root@debian:~# dpkg -i etcd_2.2.0+dfsg-2_amd64.deb
root@debian:~# systemctl stop etcd
root@debian:~# rm -rf /var/lib/etcd/default
root@debian:~# vim /etc/default/etcd
+------------------------------------------------------------------------------------------------------------------------+
| ## etcd(1) daemon options                                                                                               |
| ## See "/usr/share/doc/etcd/Documentation/configuration.md.gz".                                                         |
|                                                                                                                         |
| ### Member Flags                                                                                                        |
|                                                                                                                         |
| ##### -name                                                                                                             |
| ## Human-readable name for this member.                                                                                 |
| ## default: host name returned by `hostname`.                                                                           |
| ## This value is referenced as this node's own entries listed in the `-initial-cluster`                                 |
| ## flag (Ex: `default=http://localhost:2380` or `default=http://localhost:2380,default=http://localhost:7001`).         |
| ## This needs to match the key used in the flag if you're using [static boostrapping](clustering.md#static).            |
| # ETCD_NAME="hostname"                                                                                                  |
| ETCD_NAME="infra0"                                                                                                      |
|                                                                                                                         |
| ##### -data-dir                                                                                                         |
| ## Path to the data directory.                                                                                          |
| # ETCD_DATA_DIR="/var/lib/etcd/default"                                                                                 |
|                                                                                                                         |
| ##### -wal-dir                                                                                                          |
| ## Path to the dedicated wal directory. If this flag is set, etcd will write the                                        |
| ## WAL files to the walDir rather than the dataDir. This allows a dedicated disk                                        |
| ## to be used, and helps avoid io competition between logging and other IO operations.                                  |
| ## default: ""                                                                                                          |
| # ETCD_WAL_DIR                                                                                                          |
|                                                                                                                         |
| ##### -snapshot-count                                                                                                   |
| ## Number of committed transactions to trigger a snapshot to disk.                                                      |
| ## default: "10000"                                                                                                     |
| # ETCD_SNAPSHOT_COUNT="10000"                                                                                           |
|                                                                                                                         |
| ##### -heartbeat-interval                                                                                               |
| ## Time (in milliseconds) of a heartbeat interval.                                                                      |
| ## default: "100"                                                                                                       |
| # ETCD_HEARTBEAT_INTERVAL="100"                                                                                         |
|                                                                                                                         |
| ##### -election-timeout                                                                                                 |
| ## Time (in milliseconds) for an election to timeout.                                                                   |
| ## See /usr/share/doc/etcd/Documentation/tuning.md                                                                      |
| ## default: "1000"                                                                                                      |
| # ETCD_ELECTION_TIMEOUT="1000"                                                                                          |
|                                                                                                                         |
| ##### -listen-peer-urls                                                                                                 |
| ## List of URLs to listen on for peer traffic. This flag tells the etcd to accept                                       |
| ## incoming requests from its peers on the specified scheme://IP:port combinations.                                     |
| ## Scheme can be either http or https. If 0.0.0.0 is specified as the IP, etcd                                          |
| ## listens to the given port on all interfaces. If an IP address is given as                                            |
| ## well as a port, etcd will listen on the given port and interface.                                                    |
| ## Multiple URLs may be used to specify a number of addresses and ports to listen on.                                   |
| ## The etcd will respond to requests from any of the listed addresses and ports.                                        |
| ## example: "http://10.0.0.1:2380"                                                                                      |
| ## invalid example: "http://example.com:2380" (domain name is invalid for binding)                                      |
| ## default: "http://localhost:2380,http://localhost:7001"                                                               |
| # ETCD_LISTEN_PEER_URLS="http://localhost:2380,http://localhost:7001"                                                   |
| ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380,http://0.0.0.0:7001"                                                         |
|                                                                                                                         |
| ##### -listen-client-urls                                                                                               |
| ## List of URLs to listen on for client traffic. This flag tells the etcd to accept                                     |
| ## incoming requests from the clients on the specified scheme://IP:port combinations.                                   |
| ## Scheme can be either http or https. If 0.0.0.0 is specified as the IP, etcd                                          |
| ## listens to the given port on all interfaces. If an IP address is given as                                            |
| ## well as a port, etcd will listen on the given port and interface.                                                    |
| ## Multiple URLs may be used to specify a number of addresses and ports to listen on.                                   |
| ## The etcd will respond to requests from any of the listed addresses and ports.                                        |
| ## (ADVERTISE_CLIENT_URLS is required when LISTEN_CLIENT_URLS is set explicitly).                                       |
| ## example: "http://10.0.0.1:2379"                                                                                      |
| ## invalid example: "http://example.com:2379" (domain name is invalid for binding)                                      |
| ## default: "http://localhost:2379,http://localhost:4001"                                                               |
| # ETCD_LISTEN_CLIENT_URLS="http://localhost:2379,http://localhost:4001"                                                 |
| ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379,http://0.0.0.0:4001"                                                       |
|                                                                                                                         |
| ##### -max-snapshots                                                                                                    |
| ## Maximum number of snapshot files to retain (0 is unlimited)                                                          |
| ## default: 5                                                                                                           |
| # ETCD_MAX_SNAPSHOTS="5"                                                                                                |
|                                                                                                                         |
| ##### -max-wals                                                                                                         |
| ## Maximum number of wal files to retain (0 is unlimited)                                                               |
| ## default: 5                                                                                                           |
| # ETCD_MAX_WALS="5"                                                                                                     |
|                                                                                                                         |
| ##### -cors                                                                                                             |
| ## Comma-separated whitelist of origins for CORS (cross-origin resource sharing).                                       |
| ## default: none                                                                                                        |
| # ETCD_CORS                                                                                                             |
|                                                                                                                         |
| ### Clustering Flags                                                                                                    |
| ## For an explanation of the various ways to do cluster setup, see:                                                     |
| ## /usr/share/doc/etcd/Documentation/clustering.md.gz                                                                   |
| ##                                                                                                                      |
| ## The command line parameters starting with -initial-cluster will be                                                   |
| ## ignored on subsequent runs of etcd as they are used only during initial                                              |
| ## bootstrap process.                                                                                                   |
|                                                                                                                         |
| ##### -initial-advertise-peer-urls                                                                                      |
| ## List of this member's peer URLs to advertise to the rest of the cluster.                                             |
| ## These addresses are used for communicating etcd data around the cluster.                                             |
| ## At least one must be routable to all cluster members.                                                                |
| ## These URLs can contain domain names.                                                                                 |
| ## example: "http://example.com:2380, http://10.0.0.1:2380"                                                             |
| ## default: "http://localhost:2380,http://localhost:7001"                                                               |
| # ETCD_INITIAL_ADVERTISE_PEER_URLS="http://localhost:2380,http://localhost:7001"                                        |
| ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.16.2.163:2380,http://172.16.2.163:7001"                                    |
|                                                                                                                         |
| ##### -initial-cluster                                                                                                  |
| ## initial cluster configuration for bootstrapping.                                                                     |
| ## The key is the value of the `-name` flag for each node provided.                                                     |
| ## The default uses `default` for the key because this is the default for the `-name` flag.                             |
| ## default: "default=http://localhost:2380,default=http://localhost:7001"                                               |
| # ETCD_INITIAL_CLUSTER="default=http://localhost:2380,default=http://localhost:7001"                                    |
| ETCD_INITIAL_CLUSTER="infra0=http://172.16.2.163:2380,infra1=http://172.16.2.164:2380,infra2=http://172.16.2.165:2380"  |
|                                                                                                                         |
| ##### -initial-cluster-state                                                                                            |
| ## Initial cluster state ("new" or "existing"). Set to `new` for all members                                            |
| ## present during initial static or DNS bootstrapping. If this option is set to                                         |
| ## `existing`, etcd will attempt to join the existing cluster. If the wrong                                             |
| ## value is set, etcd will attempt to start but fail safely.                                                            |
| ## default: "new"                                                                                                       |
| # ETCD_INITIAL_CLUSTER_STATE="existing"                                                                                 |
| ETCD_INITIAL_CLUSTER_STATE="new"                                                                                        |
|                                                                                                                         |
| ##### -initial-cluster-token                                                                                            |
| ## Initial cluster token for the etcd cluster during bootstrap.                                                         |
| ## If you are spinning up multiple clusters (or creating and destroying a                                               |
| ## single cluster) with same configuration for testing purpose, it is highly                                            |
| ## recommended that you specify a unique initial-cluster-token for the                                                  |
| ## different clusters.                                                                                                  |
| ## default: "etcd-cluster"                                                                                              |
| # ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"                                                                             |
| ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"                                                                             |
|                                                                                                                         |
| ##### -advertise-client-urls                                                                                            |
| ## List of this member's client URLs to advertise to the rest of the cluster.                                           |
| ## These URLs can contain domain names.                                                                                 |
| ## example: "http://example.com:2379, http://10.0.0.1:2379"                                                             |
| ## Be careful if you are advertising URLs such as http://localhost:2379 from a                                          |
| ## cluster member and are using the proxy feature of etcd. This will cause loops,                                       |
| ## because the proxy will be forwarding requests to itself until its resources                                          |
| ## (memory, file descriptors) are eventually depleted.                                                                  |
| ## default: "http://localhost:2379,http://localhost:4001"                                                               |
| # ETCD_ADVERTISE_CLIENT_URLS="http://localhost:2379,http://localhost:4001"                                              |
| ETCD_ADVERTISE_CLIENT_URLS="http://172.16.2.163:2379,http://172.16.2.163:4001"                                                    |
|                                                                                                                         |
| ##### -discovery                                                                                                        |
|  ## Discovery URL used to bootstrap the cluster.                                                                         |
| ## default: none                                                                                                        |
| # ETCD_DISCOVERY                                                                                                        |
|                                                                                                                         |
| ##### -discovery-srv                                                                                                    |
| ## DNS srv domain used to bootstrap the cluster.                                                                        |
| ## default: none                                                                                                        |
| # ETCD_DISCOVERY_SRV                                                                                                    |
|                                                                                                                         |
| ##### -discovery-fallback                                                                                               |
| ## Expected behavior ("exit" or "proxy") when discovery services fails.                                                 |
| ## default: "proxy"                                                                                                     |
| # ETCD_DISCOVERY_FALLBACK="proxy"                                                                                       |
|                                                                                                                         |
| ##### -discovery-proxy                                                                                                  |
| ## HTTP proxy to use for traffic to discovery service.                                                                  |
| ## default: none                                                                                                        |
| # ETCD_DISCOVERY_PROXY                                                                                                  |
|                                                                                                                         |
| ### Proxy Flags                                                                                                         |
|                                                                                                                         |
| ##### -proxy                                                                                                            |
| ## Proxy mode setting ("off", "readonly" or "on").                                                                      |
| ## default: "off"                                                                                                       |
| # ETCD_PROXY="on"                                                                                                       |
|                                                                                                                         |
| ##### -proxy-failure-wait                                                                                               |
| ## Time (in milliseconds) an endpoint will be held in a failed state before being                                       |
| ## reconsidered for proxied requests.                                                                                   |
| ## default: 5000                                                                                                        |
| # ETCD_PROXY_FAILURE_WAIT="5000"                                                                                        |
|                                                                                                                         |
| ##### -proxy-refresh-interval                                                                                           |
| ## Time (in milliseconds) of the endpoints refresh interval.                                                            |
| ## default: 30000                                                                                                       |
| # ETCD_PROXY_REFRESH_INTERVAL="30000"                                                                                   |
|                                                                                                                         |
| ##### -proxy-dial-timeout                                                                                               |
| ## Time (in milliseconds) for a dial to timeout or 0 to disable the timeout                                             |
| ## default: 1000                                                                                                        |
| # ETCD_PROXY_DIAL_TIMEOUT="1000"                                                                                        |
|                                                                                                                         |
| ##### -proxy-write-timeout                                                                                              |
| ## Time (in milliseconds) for a write to timeout or 0 to disable the timeout.                                           |
| ## default: 5000                                                                                                        |
| # ETCD_PROXY_WRITE_TIMEOUT="5000"                                                                                       |
|                                                                                                                         |
| ##### -proxy-read-timeout                                                                                               |
| ## Time (in milliseconds) for a read to timeout or 0 to disable the timeout.                                            |
| ## Don't change this value if you use watches because they are using long polling requests.                             |
| ## default: 0                                                                                                           |
| # ETCD_PROXY_READ_TIMEOUT="0"                                                                                           |
|                                                                                                                         |
| ### Security Flags                                                                                                      |
|                                                                                                                         |
| ##### -ca-file [DEPRECATED]                                                                                             |
| ## Path to the client server TLS CA file.                                                                               |
| ## default: none                                                                                                        |
| # ETCD_CA_FILE=""                                                                                                       |
|                                                                                                                         |
| ##### -cert-file                                                                                                        |
| ## Path to the client server TLS cert file.                                                                             |
| ## default: none                                                                                                        |
| # ETCD_CERT_FILE=""                                                                                                     |
|                                                                                                                         |
| ##### -key-file                                                                                                         |
| ## Path to the client server TLS key file.                                                                              |
| ## default: none                                                                                                        |
| # ETCD_KEY_FILE=""                                                                                                      |
|                                                                                                                         |
| ##### -client-cert-auth                                                                                                 |
| ## Enable client cert authentication.                                                                                   |
| ## default: false                                                                                                       |
| # ETCD_CLIENT_CERT_AUTH                                                                                                 |
|                                                                                                                         |
| ##### -trusted-ca-file                                                                                                  |
| ## Path to the client server TLS trusted CA key file.                                                                   |
| ## default: none                                                                                                        |
| # ETCD_TRUSTED_CA_FILE                                                                                                  |
|                                                                                                                         |
| ##### -peer-ca-file [DEPRECATED]                                                                                        |
| ## Path to the peer server TLS CA file. `-peer-ca-file ca.crt` could be replaced                                        |
| ## by `-peer-trusted-ca-file ca.crt -peer-client-cert-auth` and etcd will perform the same.                             |
| ## default: none                                                                                                        |
| # ETCD_PEER_CA_FILE                                                                                                     |
|                                                                                                                         |
| ##### -peer-cert-file                                                                                                   |
| ## Path to the peer server TLS cert file.                                                                               |
| ## default: none                                                                                                        |
| # ETCD_PEER_CERT_FILE                                                                                                   |
|                                                                                                                         |
| ##### -peer-key-file                                                                                                    |
| ## Path to the peer server TLS key file.                                                                                |
| ## default: none                                                                                                        |
| # ETCD_PEER_KEY_FILE                                                                                                    |
|                                                                                                                         |
| ##### -peer-client-cert-auth                                                                                            |
| ## Enable peer client cert authentication.                                                                              |
| ## default: false                                                                                                       |
| # ETCD_PEER_CLIENT_CERT_AUTH                                                                                            |
|                                                                                                                         |
| ##### -peer-trusted-ca-file                                                                                             |
| ## Path to the peer server TLS trusted CA file.                                                                         |
| ## default: none                                                                                                        |
| # ETCD_PEER_TRUSTED_CA_FILE                                                                                             |
|                                                                                                                         |
| ### Logging Flags                                                                                                       |
| ##### -debug                                                                                                            |
| ## Drop the default log level to DEBUG for all subpackages.                                                             |
| ## default: false (INFO for all packages)                                                                               |
| # ETCD_DEBUG                                                                                                            |
|                                                                                                                         |
| ##### -log-package-levels                                                                                               |
| ## Set individual etcd subpackages to specific log levels.                                                              |
| ## An example being `etcdserver=WARNING,security=DEBUG`                                                                 |
| ## default: none (INFO for all packages)                                                                                |
| # ETCD_LOG_PACKAGE_LEVELS                                                                                               |
|                                                                                                                         |
|                                                                                                                         |
| #### Daemon parameters:                                                                                                 |
| # DAEMON_ARGS=""                                                                                                        |
|                                                                                                                         |
+------------------------------------------------------------------------------------------------------------------------+
:wq
root@debian:~# 
root@debian:~# systemctl start etcd
root@debian:~# systemctl status etcd

关键字:Debian、VPN、PPTP 时间:2016年5月

0.前言

本文于2016年8月编写。转载请注明源于www.fish2bird.com

1.环境

操作系统为Debian 8。

2.步骤

a)安装并设置pptp

root@debian:/# apt-get install pptpd
...
root@debian:/# cp /etc/pptpd.conf /etc/pptpd.conf.old
root@debian:/# echo "localip 192.168.15.1" >> /etc/pptpd.conf
root@debian:/# echo "remoteip 192.168.15.100-150" >> /etc/pptpd.conf
root@debian:/# service pptpd restart

b)打开网卡转发

root@debian:/# vim /etc/sysctl.conf
...
#net.ipv4.ip_forward=1
net.ipv4.ip_forward=1 
...
:wq
root@debian:/# sysctl -p

c)设置转发规则

root@debian:/# iptables -t nat -A POSTROUTING -s 192.168.15.0/24 -o eth0 -j MASQUERADE

d)设置DNS

root@debian:/# vim /etc/ppp/pptpd-options
...
#ms-dns 10.0.0.1
#ms-dns 10.0.0.2
ms-dns 8.8.8.8
ms-dns 8.8.4.4
...
:wq
root@debian:/#

e)添加VPN用户密码

root@debian:/# vim /etc/ppp/chap-secrets
# Secrets for authentication using CHAP
# client        server  secret                  IP addresses
user1           pptpd   123456                  *
:wq
root@debian:/#

0.前言

本文于2016年12月编写。转载请注明源于www.fish2bird.com

1.接口

http://qt.gtimg.cn/q=sz000001

v_sz000001="51~平安银行~000001~9.52~9.48~9.50~671452~320552~350900~9.51~2107~9.50~4457~9.49~2416~9.48~1292~9.47~864~9.52~3203~9.53~6722~9.54~8049~9.55~24167~9.56~8702~15:00:03/9.52/7563/B/7199976/14926|14:57:00/9.50/108/S/102600/14837|14:56:57/9.51/39/B/37089/14835|14:56:54/9.51/55/B/52305/14831|14:56:51/9.51/2/B/1902/14827|14:56:48/9.51/2/B/1902/14824~20161208150133~0.04~0.42~9.55~9.43~9.50/663889/630858528~671452~63806~0.46~7.16~~9.55~9.43~1.27~1392.89~1634.62~0.92~10.43~8.53~1.01";

http://qt.gtimg.cn/q=sh600004

v_sh600004="1~白云机场~600004~14.64~14.60~14.62~55830~24874~30956~14.63~39~14.62~1722~14.61~95~14.60~496~14.59~86~14.64~57~14.65~539~14.66~201~14.67~660~14.68~604~14:59:52/14.64/3/B/4392/24876|14:59:46/14.64/10/B/14640/24870|14:59:17/14.64/5/B/7320/24820|14:58:37/14.64/10/B/14640/24756|14:58:16/14.63/6/S/8778/24721|14:57:56/14.64/2/B/2928/24683~20161208150857~0.04~0.27~14.70~14.57~14.64/55812/81726910~55830~8175~0.49~12.17~~14.70~14.57~0.89~168.36~168.36~1.58~16.06~13.14~0.99";

2.准备

MySQL数据库建表

CREATE TABLE `stock` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `c0` varchar(8) DEFAULT NULL COMMENT '未知',
  `c1` varchar(24) DEFAULT NULL COMMENT '名字',
  `c2` varchar(8) DEFAULT NULL COMMENT '代码',
  `c3` double DEFAULT NULL COMMENT '当前价格',
  `c4` double DEFAULT NULL COMMENT '昨收',
  `c5` double DEFAULT NULL COMMENT '今开',
  `c6` int(11) DEFAULT NULL COMMENT '成交量(手)',
  `c7` int(11) DEFAULT NULL COMMENT '外盘',
  `c8` int(11) DEFAULT NULL COMMENT '内盘',
  `c9` double DEFAULT NULL COMMENT '买一',
  `c10` int(11) DEFAULT NULL COMMENT '买一量(手)',
  `c11` double DEFAULT NULL COMMENT '买二',
  `c12` int(11) DEFAULT NULL COMMENT '买二量(手)',
  `c13` double DEFAULT NULL COMMENT '买三',
  `c14` int(11) DEFAULT NULL COMMENT '买三量(手)',
  `c15` double DEFAULT NULL COMMENT '买四',
  `c16` int(11) DEFAULT NULL COMMENT '买四量(手)',
  `c17` double DEFAULT NULL COMMENT '买五',
  `c18` int(11) DEFAULT NULL COMMENT '买五量(手)',
  `c19` double DEFAULT NULL COMMENT '卖一',
  `c20` int(11) DEFAULT NULL COMMENT '卖一量',
  `c21` double DEFAULT NULL COMMENT '卖二',
  `c22` int(11) DEFAULT NULL COMMENT '卖二量(手)',
  `c23` double DEFAULT NULL COMMENT '卖三',
  `c24` int(11) DEFAULT NULL COMMENT '卖三量(手)',
  `c25` double DEFAULT NULL COMMENT '卖四',
  `c26` int(11) DEFAULT NULL COMMENT '卖四量(手)',
  `c27` double DEFAULT NULL COMMENT '卖五',
  `c28` int(11) DEFAULT NULL COMMENT '卖五量(手)',
  `c29` varchar(256) DEFAULT NULL COMMENT '最近逐笔成交',
  `c30` varchar(16) DEFAULT NULL COMMENT '时间',
  `c31` double DEFAULT NULL COMMENT '涨跌',
  `c32` double DEFAULT NULL COMMENT '涨跌%',
  `c33` double DEFAULT NULL COMMENT '最高',
  `c34` double DEFAULT NULL COMMENT '最低',
  `c35` varchar(32) DEFAULT NULL COMMENT '价格/成交量(手)/成交额',
  `c36` int(11) DEFAULT NULL COMMENT '成交量(手)',
  `c37` int(11) DEFAULT NULL COMMENT '成交额(万)',
  `c38` int(11) DEFAULT NULL COMMENT '换手率',
  `c39` double DEFAULT NULL,
  `c40` varchar(16) DEFAULT NULL,
  `c41` double DEFAULT NULL COMMENT '最高',
  `c42` double DEFAULT NULL COMMENT '最低',
  `c43` double DEFAULT NULL COMMENT '振幅',
  `c44` double DEFAULT NULL COMMENT '流通市值',
  `c45` double DEFAULT NULL COMMENT '总市值',
  `c46` double DEFAULT NULL COMMENT '市净率',
  `c47` double DEFAULT NULL COMMENT '涨停价',
  `c48` double DEFAULT NULL COMMENT '跌停价',
  `c49` double DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=507 DEFAULT CHARSET=utf8mb4;

3.python代码

#
import MySQLdb
import re
import urllib2

conn = MySQLdb.connect(
	host='local',
	port=3306,
	user='root',
	passwd='123456',
	db='test',
	charset='utf8',
	)

cur = conn.cursor()

for i in range(1,3000):
	url = 'http://qt.gtimg.cn/q=sz%06d' % i
	print url,
	req = urllib2.Request(url)
	response = urllib2.urlopen(req)
	content = response.read().decode('GBK')
	row = re.findall('^v_(sh|sz)\d{6}\="(.*)";', content)
	if len(row) == 1:
		cols = row[0][1].split('~')
		if cols[39]=='':
			cols[39]=None
			print 'c39=null',
		try:
			cur.execute("INSERT INTO stock (c0,c1,c2,c3,c4,c5,c6,c7,c8,c9,\
			c10,c11,c12,c13,c14,c15,c16,c17,c18,c19,\
			c20,c21,c22,c23,c24,c25,c26,c27,c28,c29,\
			c30,c31,c32,c33,c34,c35,c36,c37,c38,c39,\
			c40,c41,c42,c43,c44,c45,c46,c47,c48,c49)\
			VALUES (\
			%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,\
			%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,\
			%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,\
			%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,\
			%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)", cols)
			print '...[OK]' 
		except MySQLdb.Error,e:
			print "%d %s", e.args[0], e.args[1]
			print sql
	else:
		print content,
		print '...[Failed]'
	if i%50 == 0:
		conn.commit()
		print 'Commit per 100 result.'
cur.close()
conn.close()

2016年12月

shell命令

root@debian:~# mysqlimport -uroot -p123456 --fields-terminated-by=',' --fields-enclosed-by='"' --fields-escaped-by='"' database_name /var/lib/mysql-files/table_name.csv

参数说明:
–fields-terminated-by,字段间的分割符。
–fields-enclosed-by,单字段的前后分界限符。
–fields-escaped-by,转义字符。
database_name,库名。
table_name,即是文件名又是表名,必须一致。

常见问题

mysqlimport: Error: 1290, The MySQL server is running with the --secure-file-priv option so it cannot execute this statement, when using table: ...

MySQL启动参数–secure-file-priv,指定了一个目录,默认是/var/lib/mysql-files/,要导入的文件必须放在此目录下,否则在会出现如上错误。

0.前言

本文于2016年8月编写。

转载请注明源于www.fish2bird.com

1.准备工作

服务器一,操作系统Debian 8,IP地址192.168.15.10,主机名master,mysql版本5.5。

服务器二,操作系统Debian 8,IP地址192.168.15.11,主机名slave,mysql版本5.5。

2.配置

a)配置Master

root@master:/# vim /etc/mysql/my.cnf
...
server-id               = 1
log_bin                 = /var/log/mysql/mysql-bin.log
#binlog_do_db           = include_database_name
#binlog_ignore_db       = include_database_name
...
:wq
root@master:/# service mysql restart
root@master:/# mysql -uroot -p
mysql> GRANT REPLICATION SLAVE ON *.* to 'slave001'@'%' identified by '123456';
mysql> flush privileges;
mysql> flush tabels with read lock;
mysql> show master status;
+------------------+----------+--------------+------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000001 |      108 |              |                  |
+------------------+----------+--------------+------------------+
mysql>

此时不要退出,等从库设置并启动完成后还需要执行unlock tables。

mysql> unlock tables;

b)配置slave

root@slave:/# vim /etc/mysql/my.cnf
...
server-id               = 2
log_bin                 = /var/log/mysql/mysql-bin.log
#binlog_do_db           = include_database_name
#binlog_ignore_db       = include_database_name
...
:wq
root@slave:/# service mysql restart
root@slave:/# mysqldump -h192.168.15.10 -uroot -p –-all-databases –-add-drop-database –-add-drop-table > dump.sql
root@slave:/# mysql -uroot -p < dump.sql
root@slave:/# mysql -uroot -p
mysql> change master to master_host='192.168.15.10',master_user='slave001',master_password='123456',master_log_file='mysql-bin.000001',master_log_pos=108;
mysql> start slave;
mysql> show slave status\G
...

确认从库正常启动后,回到主库执行unlock tables。

3.验证

在master上创建test2库。

root@master:/# mysql -uroot -p
mysql> create database test2;

在slave上查看test2库是否存在。

root@slave:/# mysql -uroot -p
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| test2              |
+--------------------+

4.常见问题

主库同步数据给从库时如果发生数据不合法情况,从库同步线程会自动终止。这时需要解决不合法问题后再start slave,通过日志可以发现该问题。

0.前言

本文于2015年10月编写。

转载请注明源于www.fish2bird.com

1.包安装

root@debian:/# apt-get install mysql-server-5.5

2.修改配置文件

root@debian:/# vim /etc/mysql/my.cnf
...
#bind-address = 127.0.0.1
...
root@debian:/# service mysql restart

设置远程访问

root@debian:/# mysql -uroot -p
Enter password:
...
mysql> use mysql;
Database changed
mysql> insert into user (Host,User,Password,Select_priv,Insert_priv,Update_priv,Delete_priv,Create_priv,Drop_priv,Reload_priv,Shutdown_priv,Process_priv,File_priv,Grant_priv,References_priv,Index_priv,Alter_priv,Show_db_priv,Super_priv,Create_tmp_table_priv,Lock_tables_priv,Execute_priv,Repl_slave_priv,Repl_client_priv,Create_view_priv,Show_view_priv,Create_routine_priv,Alter_routine_priv,Create_user_priv,Event_priv,Trigger_priv,Create_tablespace_priv,ssl_type,ssl_cipher,x509_issuer,x509_subject,max_questions,max_updates,max_connections,max_user_connections,plugin,authentication_string) select '%',User,Password,Select_priv,Insert_priv,Update_priv,Delete_priv,Create_priv,Drop_priv,Reload_priv,Shutdown_priv,Process_priv,File_priv,Grant_priv,References_priv,Index_priv,Alter_priv,Show_db_priv,Super_priv,Create_tmp_table_priv,Lock_tables_priv,Execute_priv,Repl_slave_priv,Repl_client_priv,Create_view_priv,Show_view_priv,Create_routine_priv,Alter_routine_priv,Create_user_priv,Event_priv,Trigger_priv,Create_tablespace_priv,ssl_type,ssl_cipher,x509_issuer,x509_subject,max_questions,max_updates,max_connections,max_user_connections,plugin,authentication_string from user where host='localhost' and user='root';
Query OK, 1 row affected (0.00 sec)
Records: 1  Duplicates: 0  Warnings: 0

mysql> flush privileges;
Query OK, 0 rows affected (0.01 sec)

“如果我最初问消费者他们想要什么,他们应该是会告诉我,‘要一匹更快的马!’”  — 亨利福特

福特:“你为什么需要一匹更快的马?”
客户:“因为可以跑得更快!”
福特:“你为什么需要跑得更快?”
客户:“因为这样我就可以更早的到达目的地。”
福特:“所以,你要一匹更快的马的真正用意是?”
客户:“用更短的时间、更快地到达目的地!”

0.前言

本文于2016年7月编写。

转载请注明源于www.fish2bird.com

1.介绍

sklearn,Python的机器学习算法库,目前是Python做机器学习最常用的库。它基于NumPy、SciPy和matplotlib等实现的。本文主要用到它的逻辑回归模型和评价报表功能。

pandas,Python的数据解析工具,功能强大使用广泛。它基于NumPy实现。本文主要用到它的csv文件载入功能。

iris,一种花,有较多种类。有人提供了关于这种花的数据用于分类模型的学习,这份数据已近成为非常经典且简单的多分类实验数据。

2.使用sklearn库自带的数据进行逻辑回归分析

使用红色字体标注的为关键代码

					
# iris.py
# Copyright (c) 2016 WU PENG

from sklearn.cross_validation import train_test_split
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report

# Iris is flower. The function load_iris() provides 150 records with feature and target about iris.
print "\nLoading data..."
iris = load_iris()
print "Data shape: ", iris.data.shape, iris.target.shape

# The function train_test_split() splits the data.
# By default 75% of the data is used to train, and 25% of the data is used to test.
print "\nSpliting data..."
X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target)
print "Train data shape: ", X_train.shape, y_train.shape
print "Test data shape: ", X_test.shape, y_test.shape

print "\nTraining..."
clf = LogisticRegression()
clf.fit(X_train, y_train)
print "intercept: \n", clf.intercept_
print "coef: \n", clf.coef_

print "\nTesting..."
y_true = y_test
y_pred = clf.predict(X_test)
print "true: ", y_true
print "pred: ", y_pred
target_names = ['class 0', 'class 1', 'class 2']
print classification_report(y_true, y_pred, target_names=target_names)

print "\nPredict..."
X_pred = [5, 3, 5, 2.5]
y_pred = clf.predict(X_pred)
print X_pred," = ", y_pred

3.使用csv格式数据进行逻辑回归分析

3.1.文件

lr/
lr/data.csv
lr/logistic.py

3.2.数据文件

文件名为data.csv,数据与load_iris()相同。

x1、x2、x3、x4是特征,y是目标。

x1,x2,x3,x4,y
5.1,3.5,1.4,0.2,0
4.9,3.0,1.4,0.2,0
4.7,3.2,1.3,0.2,0
4.6,3.1,1.5,0.2,0
5.0,3.6,1.4,0.2,0
5.4,3.9,1.7,0.4,0
4.6,3.4,1.4,0.3,0
5.0,3.4,1.5,0.2,0
4.4,2.9,1.4,0.2,0
4.9,3.1,1.5,0.1,0
5.4,3.7,1.5,0.2,0
4.8,3.4,1.6,0.2,0
4.8,3.0,1.4,0.1,0
4.3,3.0,1.1,0.1,0
5.8,4.0,1.2,0.2,0
5.7,4.4,1.5,0.4,0
5.4,3.9,1.3,0.4,0
5.1,3.5,1.4,0.3,0
5.7,3.8,1.7,0.3,0
5.1,3.8,1.5,0.3,0
5.4,3.4,1.7,0.2,0
5.1,3.7,1.5,0.4,0
4.6,3.6,1.0,0.2,0
5.1,3.3,1.7,0.5,0
4.8,3.4,1.9,0.2,0
5.0,3.0,1.6,0.2,0
5.0,3.4,1.6,0.4,0
5.2,3.5,1.5,0.2,0
5.2,3.4,1.4,0.2,0
4.7,3.2,1.6,0.2,0
4.8,3.1,1.6,0.2,0
5.4,3.4,1.5,0.4,0
5.2,4.1,1.5,0.1,0
5.5,4.2,1.4,0.2,0
4.9,3.1,1.5,0.1,0
5.0,3.2,1.2,0.2,0
5.5,3.5,1.3,0.2,0
4.9,3.1,1.5,0.1,0
4.4,3.0,1.3,0.2,0
5.1,3.4,1.5,0.2,0
5.0,3.5,1.3,0.3,0
4.5,2.3,1.3,0.3,0
4.4,3.2,1.3,0.2,0
5.0,3.5,1.6,0.6,0
5.1,3.8,1.9,0.4,0
4.8,3.0,1.4,0.3,0
5.1,3.8,1.6,0.2,0
4.6,3.2,1.4,0.2,0
5.3,3.7,1.5,0.2,0
5.0,3.3,1.4,0.2,0
7.0,3.2,4.7,1.4,1
6.4,3.2,4.5,1.5,1
6.9,3.1,4.9,1.5,1
5.5,2.3,4.0,1.3,1
6.5,2.8,4.6,1.5,1
5.7,2.8,4.5,1.3,1
6.3,3.3,4.7,1.6,1
4.9,2.4,3.3,1.0,1
6.6,2.9,4.6,1.3,1
5.2,2.7,3.9,1.4,1
5.0,2.0,3.5,1.0,1
5.9,3.0,4.2,1.5,1
6.0,2.2,4.0,1.0,1
6.1,2.9,4.7,1.4,1
5.6,2.9,3.6,1.3,1
6.7,3.1,4.4,1.4,1
5.6,3.0,4.5,1.5,1
5.8,2.7,4.1,1.0,1
6.2,2.2,4.5,1.5,1
5.6,2.5,3.9,1.1,1
5.9,3.2,4.8,1.8,1
6.1,2.8,4.0,1.3,1
6.3,2.5,4.9,1.5,1
6.1,2.8,4.7,1.2,1
6.4,2.9,4.3,1.3,1
6.6,3.0,4.4,1.4,1
6.8,2.8,4.8,1.4,1
6.7,3.0,5.0,1.7,1
6.0,2.9,4.5,1.5,1
5.7,2.6,3.5,1.0,1
5.5,2.4,3.8,1.1,1
5.5,2.4,3.7,1.0,1
5.8,2.7,3.9,1.2,1
6.0,2.7,5.1,1.6,1
5.4,3.0,4.5,1.5,1
6.0,3.4,4.5,1.6,1
6.7,3.1,4.7,1.5,1
6.3,2.3,4.4,1.3,1
5.6,3.0,4.1,1.3,1
5.5,2.5,4.0,1.3,1
5.5,2.6,4.4,1.2,1
6.1,3.0,4.6,1.4,1
5.8,2.6,4.0,1.2,1
5.0,2.3,3.3,1.0,1
5.6,2.7,4.2,1.3,1
5.7,3.0,4.2,1.2,1
5.7,2.9,4.2,1.3,1
6.2,2.9,4.3,1.3,1
5.1,2.5,3.0,1.1,1
5.7,2.8,4.1,1.3,1
6.3,3.3,6.0,2.5,2
5.8,2.7,5.1,1.9,2
7.1,3.0,5.9,2.1,2
6.3,2.9,5.6,1.8,2
6.5,3.0,5.8,2.2,2
7.6,3.0,6.6,2.1,2
4.9,2.5,4.5,1.7,2
7.3,2.9,6.3,1.8,2
6.7,2.5,5.8,1.8,2
7.2,3.6,6.1,2.5,2
6.5,3.2,5.1,2.0,2
6.4,2.7,5.3,1.9,2
6.8,3.0,5.5,2.1,2
5.7,2.5,5.0,2.0,2
5.8,2.8,5.1,2.4,2
6.4,3.2,5.3,2.3,2
6.5,3.0,5.5,1.8,2
7.7,3.8,6.7,2.2,2
7.7,2.6,6.9,2.3,2
6.0,2.2,5.0,1.5,2
6.9,3.2,5.7,2.3,2
5.6,2.8,4.9,2.0,2
7.7,2.8,6.7,2.0,2
6.3,2.7,4.9,1.8,2
6.7,3.3,5.7,2.1,2
7.2,3.2,6.0,1.8,2
6.2,2.8,4.8,1.8,2
6.1,3.0,4.9,1.8,2
6.4,2.8,5.6,2.1,2
7.2,3.0,5.8,1.6,2
7.4,2.8,6.1,1.9,2
7.9,3.8,6.4,2.0,2
6.4,2.8,5.6,2.2,2
6.3,2.8,5.1,1.5,2
6.1,2.6,5.6,1.4,2
7.7,3.0,6.1,2.3,2
6.3,3.4,5.6,2.4,2
6.4,3.1,5.5,1.8,2
6.0,3.0,4.8,1.8,2
6.9,3.1,5.4,2.1,2
6.7,3.1,5.6,2.4,2
6.9,3.1,5.1,2.3,2
5.8,2.7,5.1,1.9,2
6.8,3.2,5.9,2.3,2
6.7,3.3,5.7,2.5,2
6.7,3.0,5.2,2.3,2
6.3,2.5,5.0,1.9,2
6.5,3.0,5.2,2.0,2
6.2,3.4,5.4,2.3,2
5.9,3.0,5.1,1.8,2

3.3.代码文件

使用红色字体标注的为关键代码

# logistic.py
# Copyright (c) 2016 WU PENG

import pandas as pd
from sklearn.cross_validation import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report

print "\nLoading data..."
data = pd.read_csv('./data.csv', header=0)
feature = ['x1', 'x2', 'x3', 'x4']
target = ['y']

# The function train_test_split() splits the data.
print "\nSpliting data..."
X_train, X_test, y_train, y_test = train_test_split(data[feature], data[target])
# The function ravel() change [[0][0]...[1][1]] to [0,0,...,1,1]. Call it, otherwise there will be an error 'DataConversionWarning'. 
y_train = y_train.ravel()
y_test = y_test.ravel()
print "Train data shape: ", X_train.shape, y_train.shape
print "Test data shape: ", X_test.shape, y_test.shape

print "\nTraining..."
clf = LogisticRegression()
clf.fit(X_train, y_train)
print "intercept: \n", clf.intercept_
print "coef: \n", clf.coef_

print "\nTesting..."
y_true = y_test
y_pred = clf.predict(X_test)
print "true: ", y_true
print "pred: ", y_pred
target_names = ['class 0', 'class 1', 'class 2']
print classification_report(y_true, y_pred, target_names=target_names)

print "\nPredict..."
X_pred = [5, 3, 5, 2.5]
y_pred = clf.predict(X_pred)
print X_pred," = ", y_pred