Skip to content
Snippets Groups Projects
Commit b5b414ec authored by Patrick L.S. Connor's avatar Patrick L.S. Connor
Browse files

Initial commit

parents
Branches
No related tags found
No related merge requests found
Expand all Show whitespace changes
Inline Side-by-side
Showing
with 957 additions and 0 deletions
.ipynb_checkpoints/IPython_and_Jupyter-checkpoint.ipynb 0 → 100644
+ 957
− 0
View file @ b5b414ec
%% Cell type:markdown id: tags:
# IPython
**IPython** (Interactive Python) is an enhanced Python shell which provides a more robust and productive development environment for users. There are several key features that set it apart from the standard Python shell.
Jupyter is the web variant. It provides an interactive shell within a browser.
### History
In IPython, all your inputs and outputs are saved. There are two variables named `In` and `Out` which are assigned as you work with your results. All outputs are saved automatically to variables of the form `_N`, where `N` is the prompt number, and inputs to `_iN`. This allows you to recover quickly the result of a prior computation by referring to its number even if you forgot to store it as a variable.
%% Cell type:code id: tags:
``` python
import numpy as np
np.sin(4)**2
```
%% Cell type:code id: tags:
``` python
_1
```
%% Cell type:code id: tags:
``` python
_i1
```
%% Cell type:code id: tags:
``` python
_1 / 4.
```
%% Cell type:markdown id: tags:
### Output is asynchronous
All output is displayed asynchronously as it is generated in the Kernel. If you execute the next cell, you will see the output one piece at a time, not all at the end.
%% Cell type:code id: tags:
``` python
import time, sys
for i in range(8):
print(i)
time.sleep(0.5)
```
%% Cell type:markdown id: tags:
### Introspection
If you want details regarding the properties and functionality of any Python objects currently loaded into IPython, you can use the `?` to reveal any details that are available:
%% Cell type:code id: tags:
``` python
some_dict = {}
some_dict?
```
%% Cell type:markdown id: tags:
If available, additional detail is provided with two question marks, including the source code of the object itself.
%% Cell type:code id: tags:
``` python
from numpy.linalg import svd
svd??
```
%% Cell type:markdown id: tags:
This syntax can also be used to search namespaces with wildcards (\*).
%% Cell type:code id: tags:
``` python
%matplotlib inline
import pylab as plt
plt.*plot*?
```
%% Cell type:markdown id: tags:
### Tab completion
Because IPython allows for introspection, it is able to afford the user the ability to tab-complete commands that have been partially typed. This is done by pressing the `<tab>` key at any point during the process of typing a command.
**Place your cursor after the partially-completed command below and press tab:**
%% Cell type:code id: tags:
``` python
np.ar # hit TAB behind np.ar|
```
%% Cell type:code id: tags:
``` python
plt.hist() # hit tab within np.hist(|)
```
%% Cell type:markdown id: tags:
This can even be used to help with specifying arguments to functions, which can sometimes be difficult to remember:
%% Cell type:markdown id: tags:
### System commands
In IPython, you can type `ls` to see your files or `cd` to change directories, just like you would at a regular system prompt:
%% Cell type:code id: tags:
``` python
ls
```
%% Cell type:markdown id: tags:
Virtually any system command can be accessed by prepending `!`, which passes any subsequent command directly to the OS.
%% Cell type:code id: tags:
``` python
# !locate sklearn | grep xgboost
```
%% Cell type:markdown id: tags:
You can even use Python variables in commands sent to the OS:
%% Cell type:code id: tags:
``` python
fil = 'tensor'
#! echo $fil
!ls {fil}*.ipynb
```
%% Cell type:markdown id: tags:
The output of a system command using the exclamation point syntax can be assigned to a Python variable.
%% Cell type:code id: tags:
``` python
data_files = !ls
```
%% Cell type:code id: tags:
``` python
!ls
```
%% Cell type:code id: tags:
``` python
data_files
```
%% Cell type:markdown id: tags:
## Qt Console
If you type at the system prompt:
$ ipython qtconsole
instead of opening in a terminal, IPython will start a graphical console that at first sight appears just like a terminal, but which is in fact much more capable than a text-only terminal. This is a specialized terminal designed for interactive scientific work, and it supports full multi-line editing with color highlighting and graphical calltips for functions, it can keep multiple IPython sessions open simultaneously in tabs, and when scripts run it can display the figures inline directly in the work area.
![qtconsole](files/images/qtconsole.png)
%% Cell type:markdown id: tags:
# Jupyter Notebook
Over time, the IPython project grew to include several components, including:
* an interactive shell
* a REPL protocol
* a notebook document fromat
* a notebook document conversion tool
* a web-based notebook authoring tool
* tools for building interactive UI (widgets)
* interactive parallel Python
As each component has evolved, several had grown to the point that they warrented projects of their own. For example, pieces like the notebook and protocol are not even specific to Python. As the result, the IPython team created Project Jupyter, which is the new home of language-agnostic projects that began as part of IPython, such as the notebook in which you are reading this text.
The HTML notebook that is part of the Jupyter project supports **interactive data visualization** and easy high-performance **parallel computing**.
%% Cell type:code id: tags:
``` python
import matplotlib.pyplot as plt
#plt.style.use('fivethirtyeight')
#DK
plt.style.use('dark_background')
def f(x):
return (x-3)*(x-5)*(x-7)+85
import numpy as np
x = np.linspace(0, 10, 200)
y = f(x)
plt.plot(x,y)
```
%% Cell type:markdown id: tags:
The notebook lets you document your workflow using either HTML or Markdown.
The Jupyter Notebook consists of two related components:
* A JSON based Notebook document format for recording and distributing Python code and rich text.
* A web-based user interface for authoring and running notebook documents.
The Notebook can be used by starting the Notebook server with the command:
$ jupyter notebook
This initiates an **iPython engine**, which is a Python instance that takes Python commands over a network connection.
The **IPython controller** provides an interface for working with a set of engines, to which one or more **iPython clients** can connect.
The Notebook gives you everything that a browser gives you. For example, you can embed images, videos, or entire websites.
%% Cell type:code id: tags:
``` python
from IPython.display import HTML
HTML("<iframe src=http://pytorch.org width=700 height=350></iframe>")
```
%% Cell type:code id: tags:
``` python
from IPython.display import YouTubeVideo
YouTubeVideo("rl5DaFbLc60")
```
%% Cell type:markdown id: tags:
### Remote Code
Use `%load` to add remote code
This becomes the following cell
%% Cell type:code id: tags:
``` python
# %load http://matplotlib.org/mpl_examples/shapes_and_collections/scatter_demo.py
"""
Simple demo of a scatter plot.
"""
import numpy as np
import matplotlib.pyplot as plt
N = 50
x = np.random.rand(N)
y = np.random.rand(N)
colors = np.random.rand(N)
area = np.pi * (15 * np.random.rand(N))**2 # 0 to 15 point radii
plt.scatter(x, y, s=area, c=colors, alpha=0.5)
plt.show()
```
%% Cell type:code id: tags:
``` python
# %load http://matplotlib.org/mpl_examples/shapes_and_collections/scatter_demo.py
"""
Simple demo of a scatter plot.
"""
import numpy as np
import matplotlib.pyplot as plt
N = 50
x = np.random.rand(N)
y = np.random.rand(N)
colors = np.random.rand(N)
area = np.pi * (15 * np.random.rand(N))**2 # 0 to 15 point radii
plt.scatter(x, y, s=area, c=colors, alpha=0.5)
plt.show()
```
%% Cell type:markdown id: tags:
### Mathjax Support
Mathjax ia a javascript implementation $\alpha$ of LaTeX that allows equations to be embedded into HTML. For example, this markup:
"""$$ \int_{a}^{b} f(x)\, dx \approx \frac{1}{2} \sum_{k=1}^{N} \left( x_{k} - x_{k-1} \right) \left( f(x_{k}) + f(x_{k-1}) \right). $$"""
becomes this:
$$
\int_{a}^{b} f(x)\, dx \approx \frac{1}{2} \sum_{k=1}^{N} \left( x_{k} - x_{k-1} \right) \left( f(x_{k}) + f(x_{k-1}) \right).
$$
%% Cell type:markdown id: tags:
## SymPy Support
SymPy is a Python library for symbolic mathematics. It supports:
* polynomials
* calculus
* solving equations
* discrete math
* matrices
%% Cell type:code id: tags:
``` python
from sympy import *
init_printing()
x, y = symbols("x y")
```
%% Cell type:code id: tags:
``` python
eq = ((x+y)**2 * (x+1))
eq
```
%% Cell type:code id: tags:
``` python
expand(eq)
```
%% Cell type:code id: tags:
``` python
(1/cos(x)).series(x, 0, 6)
```
%% Cell type:code id: tags:
``` python
limit((sin(x)-x)/x**3, x, 0)
```
%% Cell type:code id: tags:
``` python
diff(cos(x**2)**2 / (1+x), x)
```
%% Cell type:markdown id: tags:
### Magic functions
IPython has a set of predefined ‘magic functions’ that you can call with a command line style syntax. These include:
* `%run`
* `%edit`
* `%debug`
* `%timeit`
* `%paste`
* `%load_ext`
%% Cell type:markdown id: tags:
Timing the execution of code; the `timeit` magic exists both in line and cell form:
%% Cell type:code id: tags:
``` python
%timeit np.linalg.eigvals(np.random.rand(100,100))
```
%% Cell type:code id: tags:
``` python
%%timeit a = np.random.rand(100, 100)
np.linalg.eigvals(a)
```
%% Cell type:code id: tags:
``` python
%%timeit
A = np.random.rand(100,100)
w = np.linalg.eig(A)
w
```
%% Cell type:markdown id: tags:
IPython also creates aliases for a few common interpreters, such as bash, ruby, perl, etc.
These are all equivalent to `%%script <name>`
%% Cell type:code id: tags:
``` python
plt.style.use('dark_background')
n=200
max=n-1
#max=n
xx = np.linspace(0,n-1,n)[:max] # consider x values 0, 1, .., 100
A_uni = np.random.rand(n,n)
A_norm = np.random.normal(loc=0,scale=1/3.14,size=(n,n))
A_lap = np.random.laplace(loc=0,scale=.707/3.14,size=(n,n))
A_exp = np.random.exponential(scale=1,size=(n,n))
#A_lap=np.sin(A_norm)
#A_exp=np.cos(A_norm)
w_uni = np.linalg.eig(A_uni)
w_norm = np.linalg.eig(A_norm)
w_lap = np.linalg.eig(A_lap)
w_exp = np.linalg.eig(A_exp)
ev_uni = np.array(np.sort(np.linalg.eigvals(A_uni)))
ev_norm = np.array(np.sort(np.linalg.eigvals(A_norm)))
ev_lap = np.array(np.sort(np.linalg.eigvals(A_lap)))
ev_exp = np.array(np.sort(np.linalg.eigvals(A_exp)))
plt.plot(xx, ev_norm[:max],label="ev. normsl")
plt.plot(xx, ev_uni[:max],label="ev. uniform ")
plt.plot(xx, ev_lap[:max],label="ev. laplace ")
plt.plot(xx, ev_exp[:max],label="ev. exponential ")
plt.legend()
plt.show()
print 'uni ',ev_uni[0],ev_uni[n-1]
print 'normal ',ev_norm[0],ev_norm[n-1]
print 'exp ',ev_exp[0],ev_exp[n-1]
print 'laplace ',ev_lap[0],ev_lap[n-1]
```
%% Cell type:code id: tags:
``` python
%timeit EigenV = np.array(np.sort(np.linalg.eigvals(np.random.exponential(scale=1,size=(1000,1000)))))
```
%% Cell type:code id: tags:
``` python
%%ruby
puts "Hello from Ruby #{RUBY_VERSION}"
```
%% Cell type:code id: tags:
``` python
%%bash
echo "hello from $BASH"
```
%% Cell type:markdown id: tags:
IPython has an `rmagic` extension that contains a some magic functions for working with R via rpy2. This extension can be loaded using the `%load_ext` magic as follows:
%% Cell type:code id: tags:
``` python
%load_ext rpy2.ipython
```
%% Cell type:markdown id: tags:
If the above generates an error, it is likely that you do not have the `rpy2` module installed. You can install this now via:
%% Cell type:code id: tags:
``` python
#!pip install rpy2
#
#!conda install rpy2
# by hand
```
%% Cell type:code id: tags:
``` python
# x,y = np.arange(10), np.random.normal(size=10)
# %R print(lm(rnorm(10)~rnorm(10)))
```
%% Cell type:code id: tags:
``` python
# %%R -i x,y -o XYcoef
# lm.fit <- lm(y~x)
# par(mfrow=c(2,2))
# print(summary(lm.fit))
# plot(lm.fit)
# XYcoef <- coef(lm.fit)
```
%% Cell type:code id: tags:
``` python
# XYcoef
```
%% Cell type:markdown id: tags:
### LaTeX
In addition to MathJax support, you may declare a LaTeX cell using the `%latex` magic:
%% Cell type:code id: tags:
``` python
%%latex
\begin{align}
\nabla \times \vec{\mathbf{B}} -\, \frac1c\, \frac{\partial\vec{\mathbf{E}}}{\partial t} & = \frac{4\pi}{c}\vec{\mathbf{j}} \\
\nabla \cdot \vec{\mathbf{E}} & = 4 \pi \rho \\
\nabla \times \vec{\mathbf{E}}\, +\, \frac1c\, \frac{\partial\vec{\mathbf{B}}}{\partial t} & = \vec{\mathbf{0}} \\
\nabla \cdot \vec{\mathbf{B}} & = 0
\end{align}
```
%% Cell type:markdown id: tags:
## Javscript
Jupyter also enables objects to declare a JavaScript representation. At first, this may seem odd as output is inherently visual and JavaScript is a programming language. However, this opens the door for rich output that leverages the full power of JavaScript and associated libraries such as D3 for output.
%% Cell type:code id: tags:
``` python
%%javascript
alert("Hello world!");
```
%% Cell type:markdown id: tags:
## Exporting and Converting Notebooks
In Jupyter, one can convert an `.ipynb` notebook document file into various static formats via the `nbconvert` tool. Currently, nbconvert is a command line tool, run as a script using Jupyter.
%% Cell type:code id: tags:
``` python
#!jupyter nbconvert --to html IPython_and_Jupyter.ipynb
```
%% Cell type:markdown id: tags:
Currently, `nbconvert` supports HTML (default), LaTeX, Markdown, reStructuredText, Python and HTML5 slides for presentations. Some types can be post-processed, such as LaTeX to PDF (this requires [Pandoc](http://johnmacfarlane.net/pandoc/) to be installed, however).
%% Cell type:markdown id: tags:
## Reproducible Research
> reproducing conclusions from a single experiment based on the measurements from that experiment
The most basic form of reproducibility is a complete description of the data and associated analyses (including code!) so the results can be *exactly* reproduced by others.
Reproducing calculations can be onerous, even with one's own work!
Scientific data are becoming larger and more complex, making simple descriptions inadequate for reproducibility. As a result, most modern research is irreproducible without tremendous effort.
*** Reproducible research is not yet part of the culture of science in general, or scientific computing in particular. ***
%% Cell type:markdown id: tags:
## Scientific Computing Workflow
There are a number of steps to scientific endeavors that involve computing:
![workflow](http://f.cl.ly/items/3B0l063n2T0H1p041U3L/workflow.png)
Many of the standard tools impose barriers between one or more of these steps. This can make it difficult to iterate, reproduce work.
The Jupyter notebook eliminates or reduces these barriers to reproducibility.
%% Cell type:markdown id: tags:
## Parallel iPython
At a high level, there are three basic components to parallel IPython:
* Engine(s) - the remote or distributed processes where your code runs.
* Client - your interface to running code on Engines.
* Controller - the collection of processes that coordinate Engines and Clients.
These components live in the `IPython.parallel` package, which has been rolled out into its own model that requires installation.
To install ipyparallel:
```bash
pip install ipyparallel
```
or via `conda`:
```bash
conda install ipyparallel
```
To install the IPython Clusters tab in Jupyter Notebook, add this to your `jupyter_notebook_config.py`:
```python
c.NotebookApp.server_extensions.append('ipyparallel.nbextension')
```
This file resides in your `~/.jupyter` subdirectory of your home directory, and should be created if it does not already exist.
%% Cell type:markdown id: tags:
Before running the next cell, make sure you have first started your cluster, you can use the [clusters tab in the dashboard](/#tab2) to do so.
%% Cell type:code id: tags:
``` python
#from ipyparallel import Client
#client = Client()
#dv = client.direct_view()
```
%% Cell type:code id: tags:
``` python
#len(dv)
```
%% Cell type:code id: tags:
``` python
#def where_am_i():
# import os
# import socket
#
# return "In process with pid {0} on host: '{1}'".format(
# os.getpid(), socket.gethostname())
```
%% Cell type:code id: tags:
``` python
#where_am_i_direct_results = dv.apply(where_am_i)
#where_am_i_direct_results.get()
```
%% Cell type:markdown id: tags:
## Links and References
%% Cell type:markdown id: tags:
[IPython Notebook Viewer](http://nbviewer.ipython.org) Displays static HTML versions of notebooks, and includes a gallery of notebook examples.
%% Cell type:markdown id: tags:
[NotebookCloud](https://notebookcloud.appspot.com) A service that allows you to launch and control IPython Notebook servers on Amazon EC2 from your browser.
%% Cell type:markdown id: tags:
[A Reference-Free Algorithm for Computational Normalization of Shotgun Sequencing Data](http://ged.msu.edu/papers/2012-diginorm/) A landmark example of reproducible research in genomics: Git repo, iPython notebook, data and scripts.
Jacques Ravel and K Eric Wommack. 2014. [All Hail Reproducibility in Microbiome Research](http://www.microbiomejournal.com/content/pdf/2049-2618-2-8.pdf). Microbiome, 2:8.
Benjamin Ragan-Kelley et al.. 2013. [Collaborative cloud-enabled tools allow rapid, reproducible biological insights](http://www.nature.com/ismej/journal/v7/n3/full/ismej2012123a.html). The ISME Journal, 7, 461–464; doi:10.1038/ismej.2012.123;
.ipynb_checkpoints/Intro_Numpy-checkpoint.ipynb 0 → 100644
+ 0
− 0
View file @ b5b414ec
This diff is collapsed.
.ipynb_checkpoints/Matplotlib_Cheatsheet-checkpoint.ipynb 0 → 100644
+ 0
− 0
View file @ b5b414ec
This diff is collapsed.
IPython_and_Jupyter.ipynb 0 → 100644
+ 0
− 0
View file @ b5b414ec
This diff is collapsed.
Intro_Numpy.ipynb 0 → 100644
+ 0
− 0
View file @ b5b414ec
This diff is collapsed.
Intro_Pandas.ipynb 0 → 100644
+ 0
− 0
View file @ b5b414ec
This diff is collapsed.
Matplotlib_Cheatsheet.ipynb 0 → 100644
+ 0
− 0
View file @ b5b414ec
This diff is collapsed.
stockholm_td_adj.dat 0 → 100644
+ 0
− 0
View file @ b5b414ec
This diff is collapsed.
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment