2015年都过了一个月了。现在才开始写新年计划有点晚了，可晚比没有强。
规划一年是个很难的事儿，所以我就先计划一小部分，并附上deadline
（deadline driven development 是最有生产力的）。
暂时就想到这么多，wilbeibi，加油！
When we talk about SSH, we are usually referring to OpenSSH - a FREE version of the SSH connectivity tools. As their official website said:
OpenSSH encrypts all traffic (including passwords) to effectively eliminate eavesdropping, connection hijacking, and other attacks. Additionally, OpenSSH provides secure tunneling capabilities and several authentication methods, and supports all SSH protocol versions.
And I believe ssh is one of the most used commands for programmers (Windows users, you have putty, that’s … not bad). In this post I am going to list some most basic usage of ssh.
They are two ways to identify users: via password and via key pair. The latter one is more secure. We can generate a key pair through:
$ ssh-keygen -t rsa -C "[email protected]"
# Creates a new ssh key, using the provided email as a label
Generating public/private rsa key pair.
# Enter file in which to save the key (/Users/you/.ssh/id_rsa): [Press enter]
where -t stands for encryption type, -C for comment. Then choose a strong passphrase (in case of your rsa keys being stolen). Now, you will see id_rsa (private key) and id_rsa.pub (public key)in your ~/.ssh/ directory(Don’t let others know your private key).
At last, add your key to ssh-agent(a keys management tool):
eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_rsa
Now it’s time use our public key. For Linux user:
ssh-copy-id user@machine
For Mac user, we can either brew install ssh-copy-id to install and use this command or typing:
cat ~/.ssh/id_rsa.pub | ssh user@machine "mkdir ~/.ssh; cat >> ~/.ssh/authorized_keys"
As you can tell, actually what we are doing here is copy the content of id_rsa.pub to server’s ~/.ssh/authorized_keys.
Laziness is a great virtue of a programmer. Add these to local ~/.ssh/config (create if not exist) to simplify your life:
Host matrix
HostName <domain name or public IP>
User <user name>
IdentityFile </path/to/private_key>
One more thing, ssh config support wildcard, so you can use
Host *compute-1.amazonaws.com
for all your ec2 instances. I also added
TCPKeepAlive=yes
ServerAliveInterval=15
ServerAliveCountMax=6
StrictHostKeyChecking=no
Compression=yes
ForwardAgent=yes
RSAAuthentication=yes
PasswordAuthentication=yes
in my config file to add more feature.
Use uname to get system information:
uname -a
-a for print all the information.
Use nproc to print the number of processing units available (GNU coreutils):
$ nproc
4
Use lscpu to display CPU architecture information (util-linux).
df is a powerful command for displaying system disk.
df -h /path/to/directory
cat /proc/partitions/ and cat /proc/mounts are also pretty handly solutions to check the partitions and mount disks.
Just as same as disk, cat /proc/meminfo could easily check memory information (Thanks to Unix’s Everything is a file design concept).
Alternatively, you can type free -m, which essentially is the same as check /meminfo. -m for display in megabytes (as you expected, -g for gigabytes, -k for kilobytes.)
last command will display user’s info like terminal, time, date and so forth. To check one specific user’s activity, last username is what you are looking for.
w is a great but rarely know command. It will display who is logged on and what they are doing. It’ll show username, terminal, from IP, login time, idle time, JCPU and the command line of their current process. If you never heard it before, I strongly suggest you to have a try.
uptime: Tell how long the system has been running.
ps: a well known command for checking current processes, for instance, to list all zombie process:
ps aux | awk '{ print $8 " " $2 }' | grep -w Z
where ps aux to show processes for all users, the process’s user, and also show the processes not attached to a terminal (check man page for more details), then awk to filter the STAT and PID field, use grep to select the line contains Z(Zombie), now we get zombie processes pids. It’s easy to kill them by kill -9 {PID}.
top/htop: Better not to use non-builtin command(for security reasons), but if you do want to, htop is a superior alternative to top – dynamically display current tasks.
To get your own public IP, both curl icanhazip.com or curl ifconfig.me are easy ways to do that(previous one is much faster).
ping: Even my mother knows to use ping to check network connectivity.
ifconfig: A frequently used tool to view network interface information. BTW, I wrote a script to filter IP, MAC addresses and networks from ifconfig (tested on Ubuntu, Fedora, OmniOS and FreeBSD).
lsof, aka list open files, is definitely a swiss army knife for analyzing network. lsof -i for list all open Internet and X.25 network files. (The examples below are from Daniel Miessler’s blog, see reference)
lsof -iTCP # Show only TCP connections
lsof -i:80 # Show networking only relate to port 80
lsof [email protected] # Show connections with particular IP
lsof -u username # Show given user's connections
lsof -u ^username # Show connections except given user
ss -s: display Currently Established, Closed, Orphaned and Waiting TCP sockets
By useradd:
useradd -m -d <HomeDir> -g <Group> username
It’s optional to specify the new user’s home directory and group, but I strongly suggest to do so. -m stands for create home, -d to allocate a directory. (Warning, don’t mess up useradd and adduser, the later one is a higher level’s implementation. Here is a detailed explanation of these two’s differences.)
By groupadd:
groupadd groupname
By usermod:
usermod -a -G
where usermod means modify a user account, -a stands for append, append this user to a group.
Well, there is not such a built-in command for that, but we can use:
grep ‘^groupname’ /etc/group
or apt-get install members, then
members groupname
Sticky bit is used for directories. As wikipedia said:
When the sticky bit is set, only the item’s owner, the directory’s owner, or root can rename or delete files. Without the sticky bit set, any user with write and execute permissions for the directory can rename or delete contained files, regardless of owner.
For example, if the professor create a /homework directory with sticky bit, every student can upload their homework, but they cannot rename or delete other students’ homework.
chmod +t /path/to/directory
or
chmod 1755
where 1 stands for sticky bit, 7 for owner has all privilege, 5 for read and execute privilege for the group, and for others.
Now, /path/to/directory should looks like this (replaced last character):
drwxr-xr-t 1 root other 0 Nov 10 12:57 test
As wikipedia said, if the sticky-bit is set on the directory without the execution bit set for the others category, it is indicated with a capital T:
drwxr-xr-T 1 root other 0 Nov 10 12:57 test
One sentence explanation: Regardless of who runs this program, run it as the user who owns it, not the user that executes it.
chmod u+s /path/to/file
For instance, a simple shell script showfile.sh has set setuid as root privilege:
#!/bin/sh
# showfile
ls -l | sort
And If I am a bad guy, I could easily write script :
rm -rf /some/where/important
and saved as name ls, add my ls to the front of $PATH. Now when I tried to run showfile.sh, Boom ! The files are deleted.
If you found grammar errors or typos, please feel free to help me correct it.
In this post I am going to show you some common scenarios you may suffer when you use git. To benefit from this tutorial please make sure you have already set-up a git environment. If not, I would advise you to take a look at here.
git add file
git commit -m "Aha, file modified"
Or, just type
git commit -am "Aha, file modified"
After that, push to remote repository:
git push origin branch_name
So, what’s difference between these two? I will reach to that later.
It’s a good practice to fix a wrong commit rather than make a new commit.
So, first, edit the file with the problem, make the corrections, then:
git add now_right_file
git commit --amend
git push --force branch_name # Warning!
Be careful, The --force is dangerous, it works fine to me for 99% cases, but it dose have potential harmness, and that’s why Linus doesn’t recommend it.
There are two ways of delete files, delete locally and commit to remote repository, or just directly delete files in remote repository, like:
git rm --cached file_to_delete
Even better, you can delete all the files match a certain glob:
git rm --cached 'merge-*' # delete all the files start with "merge-"
There already has an excellent and well accepted answer on StackOverflow, it’s way much better than my explanation –> link:
In the simplest terms,
git pulldoes agit fetchfollowed by agit merge.
You can do agit fetchat any time to update your remote-tracking branches underrefs/remotes/<remote>/. This operation never changes any of your own local branches underrefs/heads, and is safe to do without changing your working copy. I have even heard of people running git fetch periodically in a cron job in the background (although I wouldn’t recommend doing this).
A
git pullis what you would do to bring a local branch up-to-date with its remote version, while also updating your other remote-tracking branches.
git pull # will auto merge unconflicted part
git status # check the information of conflicted files
Use your favorite editor to edit the conflicted file with “<<<<<<” and “>>>>>>”, save it, commit it, that’s all.
git checkout latest_branch
git merge -s ours to_overwrite_branch
What’s the ours means here? It’s a merge strategy, you can find it in git checkout doc:
git checkout [–ours| theirs] branch
–ours
–theirs
When checking out paths from the index, check out stage #2 (ours) or #3 (theirs) for unmerged paths.
The index may contain unmerged entries because of a previous failed merge. By default, if you try to check out such an entry from the index, the checkout operation will fail and nothing will be checked out. Using -f will ignore these unmerged entries. The contents from a specific side of the merge can be checked out of the index by using –ours or –theirs. With -m, changes made to the working tree file can be discarded to re-create the original conflicted merge result.
git branch -d died_branch
git push origin --delete die_branch # or git push origin :died_branch
git reflog show # find revision hash
git checkout revision_hash .
I will explain this a little bit. git reflog show gives us a list of all the commits and their hashes. Then, checkout that specific hash.
Read more:
Many people always ask how to combine git add and git commit in one command, and the most answered solution is git commit -a -m "blah blah".
Yes and no. For the files which have been git add before, git commit -a will do the git add for you. But for rest files(aka untracked files), we have to the git add. If you really want to save the time for these tedious work, alias is what you are looking for.
In .gitignore:
# Ignore everything
*
!except_script.sh
This will ignore everything but except_scrpit.sh.
Once my silly cat was dancing on my keyboard after a commit mess up all the files!
Luckily, we can use
git reset hard --HEAD^
to revert to the previous commit.
Or, I wrongly git add should_not_add_file , we can also use
git reset HEAD should_not_add_file
to upstage that file.
Stolen from Stackoverflow again
git clone -b <branch> <remote_repo>
Example:
git clone -b my-branch [email protected]:user/myproject.git
Alternative (no public key setup needed):
git clone -b my-branch https://[email protected]/username/myproject.git
Recently, I am working on a web project collaborate with my friend Kelun. He is in charge of front-end code and I am taking care of backend code. So, here is a problem: How can we programming together smoothly? Which means, at any moment of time, I can check his code while he is programming.
No, that’s little bit heavy for this project. So, what’s alternative choices? Dropbox! Dropbox maybe the easiest way to share folder (wait, you means rsync? Dropbox did a lot of algorithm improvement to ensure it’s higher speed of syncing)
But, something weird happens. My Webstorm automatically changed layout views. That because he is also using Webstorm, and In each project, Webstorm use a .idea/ directory to save specific settings (as the document below said).
Project settings are stored with each specific project as a set of xml files under the .idea folder. If you specify the default project settings, these settings will be automatically used for each newly created project.
.gitginore like file in Dropbox?Sure. Of course it’s not as powerful as .gitginore. In dropbox -> Preference -> Account -> Change Setting, unclick .idea folder, that’s all.
Also, I strongly suggest unclick node_modules folder. It takes Dropbox too much time to synchronize a bunch of small pieces of files.
And if you sometimes use Emacs, to avoid annoying temporary files (but some time really save you ass), the only way I know is to add this in .emacs file.
(setq make-backup-files nil)
Please feel free to correct my typos or grammar.
There is frequently a need to scrape data. Obviously, Python is a good choice for this. The famous libraries like BeautifulSoup provides a bunch of functions to do these stuffs. But personally, I prefer lxml.
There already has some comparison about pros and cons of each library. As lxml document said:
BeautifulSoup uses a different parsing approach. It is not a real HTML parser but uses regular expressions to dive through tag soup. It is therefore more forgiving in some cases and less good in others. It is not uncommon that lxml/libxml2 parses and fixes broken HTML better, but BeautifulSoup has superior support for encoding detection. It very much depends on the input which parser works better.
… …
The downside of using this parser is that it is much slower than the HTML parser of lxml. So if performance matters, you might want to consider using soupparser only as a fallback for certain cases.
In short: lxml is faster when parsing well-formed web page.
This is a common scenario. First get links of each entries in a index page.
For example, find all housing in http://losangeles.craigslist.org/hhh/index.html. In Chrome, Inspect Element, get XPath link from one link:
The xpath is /*[@id="toc_rows"]/div[2]/p[1]/span[2]/a/@href, from p[1] to p[100]. Save these links to a file crag_link.txt.
from lxml import html
import requests
with open('crag_link.txt', 'a') as f:
for i in range(0, 1000, 100):
pg = 'http://losangeles.craigslist.org/hhh/index' + str(i) + '.html'
src = requests.get(pg)
if src.status_code == 404:
sys.exit(1)
tree = html.fromstring(src.text)
print 'Get page', i
for j in range(1, 100+1):
x_link = '//*[@id="toc_rows"]/div[2]/p[' + str(j) + ']/span[2]/a/@href'
links = tree.xpath(x_link)
for ln in links:
f.write( 'http://losangeles.craigslist.org' + ln + '\n')
f.close()
Click into one of the page, for instance, we want to get post id, copy xpath
like //*[@id="pagecontainer"]/section/section[2]/div[2]/p[1]. According to XPath syntax, these path add suffix /text() is what we need.
try:
post_id = tree.xpath('//*[@id="pagecontainer"]/section/section[2]/div[2]/p[1]/text()')
except:
# Handle Error
The reason we add try/catch block here is to prevent missing data. Wait a second, what if we have 30 attribute to scrape, do we need to write try/catch 30 times. Definitely no. Wrap them into a function might be a good idea. BTW, hardcode xpath into program is not a good idea, by writing a function, we can pass it as a parameter(Or even better, store attribute names and xpaths in a dictionary).
def get_attr(tree, xps):
return attr_name = tree.xpath(xps)
'''
xps_dict look like:
{'post_id':'//*<somehing>/p[1]/text()','post_time':'//*<somehing>/p[1]/text()'}
'''
for a, x in xps_dict.iteritems():
attr[a] = get_attr(tree, x)
For the Part 2, I will carry on, talk about encoding problem, prevent duplicates and so forth.
Linux内核作者Linus早就开始使用Mac了（虽然面对记者表现的很扭捏）。Gnome的作者更是理直气壮叛逃Linux投奔Mac。如你所知，Mac OSX是开发者们的最爱。因为它一方面有非常棒的用户体验，一方面是基于Unix，尽得shell的益处，又避免了Linux下种种繁琐。什么，太贵了，看看MacTalk是怎么说的。
工欲善其事，必先利其器。这里分享下我的Mac应用和配置。
In the last few weeks I led a life writing C programs 10 hours a day. It was interesting and frustrating, and I learned a lot from that. For one thing, programming is an enjoyable game to me. It drives me thinking problems, force me constructing ideas and making proper solutions. What’s more fascinating is, the machine would tell me whether my solution program is right or wrong in less than a second. The frustrating part is, it always tells me I was wrong. Truth to be told, I am not a good programmer as my friends thought. Hours and hours, I wasting time debugging my programs and finally found out I was wrong at the first thought. So, the problem is, what can I do with that?
The answer generally came to my mind: The Better Way to learn programming is to get away from it. Don’t panic, I don’t mean not programming, I mean not programming that soon. We live in an era that computers and internet are easy to access anytime and anywhere, which is lucky, or maybe unlucky to all of us. One story told by Alan Kay explained my reason:
When I was at Stanford with the AI project [in the late 1960s] one of the things we used to do every Thanksgiving is have a computer programming contest with people on research projects in the Bay area. The prize I think was a turkey.
[John] McCarthy used to make up the problems. The one year that Knuth entered this, he won both the fastest time getting the program running and he also won the fastest execution of the algorithm. He did it on the worst system with remote batch called the Wilbur system. And he basically beat the shit out of everyone.
And they asked him, “How could you possibly do this?” And he answered, “When I learned to program, you were lucky if you got five minutes with the machine a day. If you wanted to get the program going, it just had to be written right. So people just learned to program like it was carving stone. You sort of have to sidle up to it. That’s how I learned to program.”
The dean of my undergraduate CS department has similar experience. When the time they learned programming, the computer didn’t even has a BIOS. They had to use punched cards to operate the machine. Each program they ran has been wrote and reviewed many times. That generation students, have all become something in the field of academic now.
What I learned from these is, gcc -Wall -pendantic-error seems an easy and relatively friendly way to detect where were you wrong in your programs, but scratch paper and pencil would be your more loyally friend when you struggling figuring out the solution of programs.
Someone might ask, does it still worth in nowadays? As a successful example with plenty failure experience of rush to program first each time, I have to say, probably yes, but not necessary. Some of us with strong ability of abstraction may find is easy to run programs with no errors and warnings at first time. They usually adopt a top-down approach to organize their code: design structures and interfaces first, then implement the details. The classic example of this philosophy could be found in Robert Sedgewick’s <Algorithms>.
The sad fact is that most people, me included, cannot. So, close your laptop, make a cup of tea, prepare a pencil and some scratch paper, start to program with your brain.
Last but not least, learn to enjoy the life out of the computer science world would also helps you in computer science field. Like Eric S. Raymond’s words in <How to become a hacker>:
Again, to be a hacker, you have to enter the hacker mindset. There are some things you can do when you’re not at a computer that seem to help. They’re not substitutes for hacking (nothing is) but many hackers do them, and feel that they connect in some basic way with the essence of hacking.
- Learn to write your native language well. Though it’s a common stereotype that programmers can’t write, a surprising number of hackers (including all the most accomplished ones I know of) are very able writers.
- Read science fiction. Go to science fiction conventions (a good way to meet hackers and proto-hackers).
- Train in a martial-arts form. The kind of mental discipline required for martial arts seems to be similar in important ways to what hackers do. The most popular forms among hackers are definitely Asian empty-hand arts such as Tae Kwon Do, various forms of Karate, Kung Fu, Aikido, or Ju Jitsu. Western fencing and Asian sword arts also have visible followings. In places where it’s legal, pistol shooting has been rising in popularity since the late 1990s. The most hackerly martial arts are those which emphasize mental discipline, relaxed awareness, and control, rather than raw strength, athleticism, or physical toughness.
- Study an actual meditation discipline. The perennial favorite among hackers is Zen (importantly, it is possible to benefit from Zen without acquiring a religion or discarding one you already have). Other styles may work as well, but be careful to choose one that doesn’t require you to believe crazy things.
- Develop an analytical ear for music. Learn to appreciate peculiar kinds of music. Learn to play some musical instrument well, or how to sing.
- Develop your appreciation of puns and wordplay.
PS: After writing this post, I found Jeff Atwood wrote a similar article in 2007, here it is How To Become a Better Programmer by Not Programming.
Pro Git is an excellent book about Git, and these are my reading notes.
This is a brief notes about some points of Pro Git. It only contains first 3 chapters (contains most commonly used commands of Git). I will maintain this article and keep revising it. Hope it might helpful to you.
##Before Running Git
Snapshots, Not Differences
The major difference between Git and any other VCS (Subversion and friends included) is the way Git thinks about its data. Conceptually, most other systems store information as a list of file-based changes.
Nearly Every Operation Is Local
Most operations in Git only need local files and resources to operate ¡ª generally no information is needed from another computer on your network.
Git Has Integrity
Git use checksum, namely, SHA-1, as a reference.
Three States
Git has three main states that your files can
reside in: committed, modified, and staged. Committed means that the data is safely stored in your local database. Modified means that you have changed the file but have not committed it to your database yet. Staged means that you have marked a modified file in its current version to go into your next commit snapshot.
The basic Git workflow goes something like this:
##Git Basics
Clonegit clone git://github.com/USERNAME/REPO.git
Check Status
git status
Trace New Filegit add FILE (The FILE edited after git add need to redo git add to store the changes)
Gitignore
if you want to ignore some files, add thier patter to .gitignore. Like *.[oa] (generate while compiling)
Glob patterns are like simplified regular expressions that shells use. An asterisk (*) matches zero or more characters; [abc]matches any character inside the brackets (in this case a, b, or c); a question mark (?) matches a single character; and brackets enclosing characters separated by a hyphen([0-9]) matches any character between them (in this case 0 through 9) .
Diff between staged and last commit filegit diff --cached
Committing Changesgit status before git commit. add -m flag to add commit message, -v to explicit remind every place you modified, -a to automatically stage every file that is already tracked before doing the commit, letting skip the git add part.
Removerm FILE and git rm FILE to delete local files. git rm --cached FILE to remove files in repository.(Could use glob)
Movegit mv OLD NEW to change the names.
Viewing Historygit log, add -p to show differences in every commits, use -2 to see lasted 2 updates, –stat abbreviated stats, –pretty customized formats.
Undo
use git commit --amend to add forgetted files or reedit commit messages. (See Book 2.4.1)
Unstaging Staged Filegit reset HEAD FILE
Unmodifying a Modified File
use git status (See Book 2.4.3)
Fetching and Pullinggit fetch REMOTE-NAME pull down all the data from that remote project that you don’t have yet.
git fetch originfetches any new work that has been pushed to that server since you cloned (or last fetched from) it. It’s important to note that the fetch command pulls the data to your local repository and it doesn’t automatically merge it with any of your work or modify what you’re currently working on. You have to merge it manually into your work when you’re ready.
Pushgit push origin master will push local master branch to origin server.
Tagginggit tag to add tags.-a to annotated, -m add comments, git show to check version info.
Git Aliases
You can set up an alias for each command with git config. For example, using git config --global alias.ci commit to instead git commit with git ci. Useful Tips: git config --global alias.last 'log -1 HEAD' to check last commit by git last.
HEAD
In Git, this is apointer to the local branch you’re currently on.
New Branch
Because a branch in Git is in actuality a simple file that contains the 40 character SHA-1 checksum of the commit it points to, branches are cheap to create and destroy. Creating a new branch is as quick and simple as writing 41 bytes to a file (40 characters and a newline).
Basic Branching and Merging
switch to a certain branch git Check -b BRANCH-B, git merge to merge to master.
Delete Branchgit branch -d to delete branches. It could be failed if it was unmerged. Thus, you could use git branch -D BRANCH-B to force deleting it, and git branch --no-merged to check unmerged work.
Remote Branch
You’d better read the related part of this book! It is a little comfusing but important.
Rebasing
With the rebase command, you can take all the changes that were committed on one branch and replay them on another one. For example, git rebase master server checks out the topic branch (in this case,server) for you and replays it onto the base branch (master).It quite cool, read Book 3.5.1 to know more. Summed up in one line Do not rebase commits that you have pushed to a public repository.