Hacker Within

TBD

2020-12-02T00:00:00+00:00

Meeting Info

Date: Dec 2, 2020
Time: 12 pm
Location: Zoom

TBD

2020-11-11T00:00:00+00:00

Meeting Info

Date: Nov 11, 2020
Time: 12 pm
Location: Zoom

TBD

2020-10-28T00:00:00+00:00

Meeting Info

Date: Oct 28, 2020
Time: 12 pm
Location: Zoom

TBD

2020-10-14T00:00:00+00:00

Meeting Info

Date: Oct 14, 2020
Time: 12 pm
Location: Zoom

TBD

2020-09-30T00:00:00+00:00

Meeting Info

Date: Sep 30, 2020
Time: 12 pm
Location: Zoom

VS Code and Pluto.jl

2020-09-16T00:00:00+00:00

Meeting Info

Date: Sep 16, 2020
Time: 12 pm
Location: Zoom

VS Code (Josh)

Pluto.jl (Cail)

Demo materials can be found on the THW Github here.

I highly recommend checking out the Pluto.jl JuliaCon talk as well.

Fall '20 planning

2020-09-02T00:00:00+00:00

Meeting Info

Date: Sep 2, 2020
Time: 12 pm
Location: Jitsi

Fall ‘20 Planning

Notes:

How can we use the virtual format to our advantage?
- Code demos, collaborative hacking, etc.
More emphasis on short talks (or just discussion topics) that don’t require much preparation on the part of the speaker
Possible “anchor” for the semester: playing with data from UIUC’s COVID-19 Testing Dashboard
- less involved than last semester’s project, basically a fall-back hacking project when we don’t have other material to discuss
- data is exportable, and maybe we can scrape it?
- possible approaches: visualization, fitting, simulations
General proposal: 1-2 discussion topics or lightning talks per meeting, then use rest of time for hacking (COVID data?).
Switching to Zoom

Introductions to Julia and Rust

2020-04-08T00:00:00+00:00

Meeting Info

Date: Apr 8, 2020
Time: 12:00 PM
Location: Jitsi

Introduction to Julia (Cail Daley): slides and notebook

Introduction to Rust (Jon Drobny): slides

Healthcare Internet & Scientific Visualization

2020-03-25T00:00:00+00:00

Meeting Info

Date: Mar 25, 2020
Time: 12:00 PM
Location: [2100 NCSA][ncsa_map]

Cybersecurity & React

2020-03-04T00:00:00+00:00

Meeting Info

Date: Mar 04, 2020
Time: 12:00 PM
Location: [2100 NCSA][ncsa_map]

Group Project Discussion / Python Visualization

2020-02-19T00:00:00+00:00

Meeting Info

Date: Feb 19, 2020
Time: 12:00 PM
Location: [1040 NCSA][ncsa_map]

Group Project

From Josh Vita:

We’re planning to try to do a semester-long (or more?) software project together. It could be a simple app or game, or maybe even a small research project? Anything that you think would be a fun way to explore programming topics that you’re interested in (visualization, machine learning, optimization, front-end development, etc.) would be good. Our first project planning meeting will be next week, so try to brainstorm some ideas of what you’d like to make!

Python Visualization (Zeqian Li)

Spring '20 planning

2020-02-05T00:00:00+00:00

Meeting Info

Date: Feb 5, 2020
Time: Lunch 11:30, Meeting 12:00 PM
Location: 2100 NCSA

Spring ‘20 Planning

There has been a lot of interest in THW at Illinois but we need to continue to provide the resources students want to help them excel!

Questions to think about:

What has and hasn’t worked?
What topics should be covered?
- Is there interest in long-term learning?
- What kind of professional development would be helpful?
  - resume, cover letter, presentation templates?
THW Slack?
Paper-writing tools? (Mendeley, Zenodo, other citation managers, etc.)
HAL Spring 2020 training and hackathons?

(I borrowed from Spring ‘16 Planning post)

Professional Career Opportunities

2019-12-04T00:00:00+00:00

For our last meeting of the semester we will hear several people with professional careers similar to career paths our own members are interested in. We will learn about what steps they took to get to where they are and how their education and/or research helped them get to that stage. If you know anyone who would be interested in speaking, please email the Hacker Within president.

Check back closer to the meeting date for more information!

Parallelization

2019-11-20T00:00:00+00:00

Cail: Over the past two decades, graphical processing units (GPUs) have become one of the most successful tools for accelerating numerical computing. On Wednesday I’ll begin by discussing the hardware considerations that have led to the prevalence of GPUs; then we’ll turn to some of the principles of GPU programming. We’ll close by examining GPU implementations of some common algorithms like image blurring and matrix multiplication. Slides hosted here!

ECE 408 Applied Parallel Programming course website

Paul: Dask is an intuitive and flexible graph execution engine for Python. I will demonstrate how to parallelize Python analysis code by adding a few lines of dask in a way that is fully compatible with serial execution (i.e. you can turn off dask with a flag). If time permits, I will go over more advanced features of dask such as: building your own cluster and visualizing parallel execution (dask dashboard). Tutorial files are available on in this Github repository.

Code Challenges

2019-11-06T00:00:00+00:00

TBD

Check back closer to the meeting date for more information!

Data Visualization

2019-10-16T00:00:00+00:00

TBD

Check back closer to the meeting date for more information!

Reproducibility / Workflow

2019-10-02T00:00:00+00:00

TBD

Check back closer to the meeting date for more information!

Git Websites

2019-09-18T00:00:00+00:00

GitHub_tutorial

Why GitHub?

GitHub provides awesome repos awesome open source repos
Interesting, and maybe useful packages my new fav: the fuck
Reproducing research paper Neural ODE
Great tool for version control, in particullar for collaborative project lets get it started: link

Reference

git-commands

Github tutorial on local Jekyll

Research Showcase

2019-05-01T00:00:00+00:00

Research Showcase

To conclude the semester, members will show 5-10 min lightning talks giving an overview of their research, the languages and packages they use, and their code.

Python as Glue and Object Oriented Programming

2019-04-17T00:00:00+00:00

Python as Glue

Python is one of the most versatile and popular languages for scientific computing as of 2019. However, being an interpreted rather than a compiled language, basic loops in Python are abysmally slow. In this tutorial, we find present one possible cure for this slowness. Specifically, we will use Python to glue together C and Fortran backend codes with matplotlib visualization frontend scripts.

Object Oriented Programming

Object Oriented Programming (OOP) is a widely-used programming paradigm that emphasizes writing modular, flexible code. In today’s talk we’ll discuss some of the key concepts and advantages of OOP, with some concrete examples in Python.

Python Coding Challenge and Hacking Hour

2019-04-03T00:00:00+00:00

Python Coding Challenge

There is no tutorial or formal discussion scheduled for this week. Instead, there will be a coding challenge. Challenge participants will receive a special prize from GitHub! This time can also be used to discuss previous THW sessions, discuss and work on personal projects, and get advice on any programming issues.

Jupyter Lab and Advanced Data Structures

2019-03-27T00:00:00+00:00

Jupyter Lab

Do you use Jupyter notebooks? You should be using Jupyter Lab! Learn the benefits of Jupyter Lab in this presentation.

Advanced Data Structures

How’s a list different from a dictionary? When should you use them? Find out in this presentation.

Optimizing Code and Esoteric Programming Languages

2019-03-06T00:00:00+00:00

Optimizing Code

Is your code full of nested for-loops? There’s a better way! In this presentation we will learn common coding mistakes and how to fix and optimize them.

Terrible Programming Languages

Why do esoteric programming languages exist? What does it look like to use a few of them? Find out in this presentation!

HAI CAN HAS STDIO? VISIBLE "HAI WORLD!" KTHXBYE

Optimizing Python Code

2019-03-05T00:00:00+00:00

Optimizing (Python) code

By Josh Vita (jvita2@illinois.edu)

Many of us are students developing various scripts or packages (often in Python) either as the core of a thesis project or as a useful tool to help with our research. While it’s not a big deal to take a quick coffee break while your code runs for 15 minutes, this presentation is intended to help you avoid awkward conversations with your advisor because you’ve been waiting 3 days for some results.

This presentation will be broken down into 3 main parts:

Things you should always do because they’re easy and good practice
Things you will often want to do, especially in scientific applications
Things you only need to do when it’s gotta go fast

Things you should always do because they’re easy and good practice

Use list/dict/tuple comprehension
Take advantage of standard packages and pre-defined functions when possible
Use generators (where appropriate)

Each of these has it’s own purpose, but in general you can remember that:

List comprehensions should be your go-to in most cases due to speed and readability
map() is particularly handy if your function is already defined or you need to avoid memory issues
For loops are perfectly fine, especially if you’re not trying to build a list out of the results

For more details about comparing list comprehension, loops, and map(), see this stackoverflow post; the top 2 answers are particularly insightful

%%timeit

# In this case, the function is already defined
xs = range(10)
lst = [hex(x) for x in xs]

2.81 µs ± 34.3 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

%%timeit

# map() can be slightly better in this case
xs = range(10)
lst = list(map(hex, xs))

2.55 µs ± 8.58 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

%%timeit

# This time, we don't have a pre-defined function
xs = range(10)
lst = [x + 2 for x in xs]

1.65 µs ± 12.7 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

%%timeit

# map() incurs an extra overhead due to repeatedly calling the lambda
xs = range(10)
lst = list(map(f, xs))

1.64 µs ± 60.4 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

Beyond built-in functions, many standard libraries provide fast, efficient functions for some common operations. Not only will these save you in terms of development time (arguably more important than runtime), but they’ll probably perform better than I could write.

Here are some handy libraries that I use frequently:

itertools – great for permutations/combinations, and anything involving looping over things
glob – for accessing files
random – for all things related to random numbers

For a comprehensive list, check out the collection of standard libraries that come with Python 3

# getting permutations of items in a list -- stuff like this is what makes Python so fun
import itertools

list_of_chars = ['A', 'B', 'C']

# cool, he used generators, list comprehension, built-in functions [str().join()], AND standard libraries all in one example!

["".join(p) for p in itertools.permutations(list_of_chars)]

['ABC', 'ACB', 'BAC', 'BCA', 'CAB', 'CBA']

# extracting all file names with a given pattern
import glob

print(glob.glob('./glob_folder/*'))
print(glob.glob('./glob_folder/file*_input.dat'))
print(glob.glob('./glob_folder/file1_*.dat'))

['./glob_folder/file1_input.dat', './glob_folder/file1_output.dat', './glob_folder/file2_input.dat', './glob_folder/file2_output.dat', './glob_folder/file3_input.dat', './glob_folder/file3_output.dat', './glob_folder/README.txt']
['./glob_folder/file1_input.dat', './glob_folder/file2_input.dat', './glob_folder/file3_input.dat']
['./glob_folder/file1_input.dat', './glob_folder/file1_output.dat']

# sampling a list in random order
import random

random.sample(range(100), 10)

[50, 5, 69, 22, 87, 86, 29, 55, 49, 51]

Generators can be thought of as a way of generalizing the benefits mentioned for map()

In particular, generators can be good because they:

Avoid loading large lists into memory (see this portion of a Youtube video where one of the core Python developers discusses the importance of keeping calculations in L1 cache)
Can serve as simple, efficient, and readable replacements for iterators

# This code would create a disastrously large list ...

# lst = [x**x for x in range(10*100)] 

# ... but THIS code is perfectly happy!

(x**x for x in range(10*1)) # changing [] to () makes it a 'generator expression'

<generator object <genexpr> at 0x7f25ec305990>

# Here's an example of an iterator for creating a sequence of powers of two

class PowTwo:
    # code from blog post: https://www.programiz.com/python-programming/generator
    
    def __init__(self, max = 0):
        self.max = max

    def __iter__(self):
        self.n = 0
        return self

    def __next__(self):
        if self.n > self.max:
            raise StopIteration

        result = 2 ** self.n
        self.n += 1
        return result

# A generator is much simpler -- it's a function with 'yield' instead of 'return'!

def PowTwoGen(max = 0):
    n = 0
    while n < max:
        yield 2 ** n
        n += 1

# The same blog post also has a cool example of pipelining using generators

def pipeline():
    
    with open('sells.log') as file:
        pizza_col = (line[3] for line in file)
        per_hour = (int(x) for x in pizza_col if x != 'N/A')
        print("Total pizzas sold = ", sum(per_hour))

Things you’ll often want to do, especially in scientific applications

Vectorize your code … unless it’s too hard – if you’re working with numbers in an array, NumPy can do pretty much everything
Parallelize your code – some code is “embarrassingly parallel” and just needs some quick imports

“Vectorizing” code refers to trying to structure your code in such a way that you can perform the same operation on multiple parts of a vector at the same time. For Python, this is often translated to “stuff your data into an array, then let NumPy go nuts”.

Here are some tips if you’re trying to vectorize your code:

First, profile your code using some of the common libraries to make sure it’s worth your time – I can’t over-emphasize the importance of this
Learn about NumPy broadcasting and how to use np.newaxis
Check out the NumPy documentation for handy functions – my favorite is np.einsum
Familiarize yourself with NumPy’s Universal Functions

Below is an example from this blog post showing the benefits of profiling.

Before trying to optimize, always spend some time making sure it’s worth the effort

import numpy as np

A = np.arange(210000).reshape((50, 60, 70))
sums = np.zeros(60)

%%timeit

# loops can be horribly slow, especially for high-dimensional matrices
for j in range(A.shape[1]):
    for i in range(A.shape[0]):
        for k in range(A.shape[2]):
            sums[j] += A[i, j, k]

162 ms ± 1.58 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

%%timeit

# NumPy is way faster since it's calling optimized C libraries
sums = np.sum(A, axis=(0, 2))

312 µs ± 1.37 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

A minor point, but when I first started programming in Python I always used to make the mistake of trying to grow NumPy arrays dynamically. This is noticeably slower and can have a significant impact for large matrices.

%%timeit

# A naive example of building an array where each row is double the previous row
A = np.arange(1000).reshape((1, 1000))

for i in range(19):
    A = np.vstack([A, A[-1]*2])

393 µs ± 15.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%%timeit

# defining the full array ahead of time helps significantly
A = np.zeros(20000).reshape((20, 1000))
A[0] = np.arange(1000)

for i in range(1, 20):
    A[i, :] = A[i - 1, :]*2

113 µs ± 1.22 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

An example of vectorization in my work that groups element-wise multiplication along an axis followed by summation and an axis swap … all into one call

embedding_forces = np.einsum('pijk,pk->pji', embedding_forces, uprimes)

For many problems, one of the easiest speedups can come from simply parallelizing your code. In the case of shared-memory problems, the multiprocessing module is a godsend.

The multiprocessing module can be a quick fix when:

You need to perform the same operations on different chunks of data (SIMD)

multiprocessing can be used in a number of different cases; I won’t show an example here, but there are plenty of examples online such as this one.

Note: if you’re going to be doing lots of parallel computing in Python, you should check out this presentation by one of the major players in the Python community discussing the Global Interpreter Lock (GIL).

Things you only need to do when it’s gotta go fast

JIT compilation using Numba
Write your own libraries in C/C++, then use something like cython to call the library from python

This blog post provides some nice statistics looking at Numba performance on LU factorization

from numba import jit, void, double

def numpy_det_by_lu(y, x):
    y[0] = 1.

    N = x.shape[0]
    with np.errstate(invalid='ignore'):
        for k in range(N):
            y[0] *= x[k,k]
            xk = x[k]
            for i in range(k+1, N):
                xi = x[i]
                xi[k] /= xk[k]
                xi[k+1:] -= xi[k] * xk[k+1:]
                
fastdet_by_lu = jit(void(double[:], double[:,:]))(numba_det_by_lu)

Other useful stuff

Time-resolved memory usage using memory-profiler
Some performance metrics on why you should use ''.join() over + for string construction
Distributed-memory parallelization using mpi4py

Posting to Git and Heritage Code Review

2019-02-20T00:00:00+00:00

Posting your presentation to GitHub

A guide for uploading to the Illinois THW GitHub can be found at our GitHub page. A guide on how to edit posts using markdown can be found at How to Add Your Own Page to Our Website, courtesy of Paul Wilson from the Wisconsin chapter of THW.

Cool Links!

Data Visualization Showcase

2019-02-06T00:00:00+00:00

Data Visualization Showcase

In this meeting, Cail Daley and Yubo “Paul” Yang share how they visualize data (in and out of their own work).

Visualize 3D scatter data

presentation

example 1: rotate colored scatter

example 2:

example 3:

Kickoff Meeting

2019-01-15T00:00:00+00:00

Kickoff Meeting

In this meeting, we will be picking topics for the semester. Brainstorm topics you would like to see or present. If you need inspiration check out what other THW chapters have done in the past and our single-question survey from last semester. We can repeat topics, go more in-depth with previous topics, or go into uncharted territory!

Docker for Science

2018-12-05T00:00:00+00:00

Presentation

Docker is the most popular container manager in 2018. A container is conceptually equivalent to a virtual machine (VM). Docker can create a “container” (VM) with all dependencies for a specific app and save an “image” (equivalent to a snapshot of the VM) to an online hub. This way an application can be ran on any machine irrespective of hardware and software environment.

In this presentationslides, I will first cover the basics:

how to use existing docker images;
how to create your own image using “dockerfile”;
how to “ochestrate” multiple images using “docker-compose.yml”.

Then, I will show a practical application of docker in scientific computing. Specifically, I will compile a simulation program with non-trivial dependencies and show how it can be ran with analysis and monitoring programs on any machine using a single “docker-compose up” command.

example code

Tutorials

PyCon 2015 slides
Official Tutorial
Deploy to Amazon cloud (search “AWS Elastic Container Service”)
House-keeping tips

Data Visualization

2018-11-07T00:00:00+00:00

Data Visualization

This weeks THW meeting will give an overview of python data visualization packages and best practices for plotting using matplotlib. There will also be a chance for everyone to visualize data in a group activity. To get the most from this activity, please bring a laptop with your preferred data visualization software installed.

Automated Documentation & Software Testing

2018-10-17T00:00:00+00:00

Automated Documentation and Software Testing

In this meeting we will see a presentation on automated documentation and software testing given by Shane Keniley and Nate Walter. An automated documentation generator makes it easier for you to create a standardized document that explains how your code works. Software testing checks if your code is actually doing what you expect it to. This is especially helpful when updating, upgrading, and working collaboratively on a project.

Data Science & Machine learning

2018-10-03T00:00:00+00:00

Data Science

The first part of this weeks THW meeting will focus on data science. Together we will explore one of the most fundamental datasets in astronomy—the HR diagram—in order to show how we can learn about stellar evolution, structure, and populations from only two dimensions of data. In the process, we will touch on some of the techniques (bias identification, clustering) and coding tools (pandas, sklearn, seaborn) that are useful in data science, as well as a popular stellar evolution code, MESA.

Machine learning

The second part will explore machine learning. In this talk we’ll be doing some fun hands on machine learning (from a jupyter notebook). The activity is to get us all familiar with the data science process and tackle a classification problem. By the end of the workshop you should be able to identify a data science problem and have a feel of the procedure of solving said problem.

Intro to Python & Python Tips

2018-09-05T00:00:00+00:00

Intro to Python & Python Tips

This tutorial will give new Python users introductory skills and sharpen skills for intermediate users. We will also share tips and tricks in a group discussion.

Hardware and Software Accelerators

2018-04-18T00:00:00+00:00

Attending

<++>

Bleeding Edge Stuff

Lightning Talks

<+ person +> : <+ topic +>

Game Engines for Scientific Visualizations

2018-04-04T00:00:00+00:00

Presentation

Fermi surface from quantum Monte Carlo (QMC).
normal mode from density functional theory (DFT).
particle trajectories from molecular dynamics (MD).

Tutorials

lighting the scene
throw some flaming chairs
make tables rain from the sky
explode some cows
make birds fly

State of Unreal

Make/CMake and git/Github

2018-03-07T00:00:00+00:00

Aaron Anderson

Make/CMake

Resources:

git/Github

git and Github tutorial for editing, tracking, and hosting the THW-IL website.

Follow-up to question about cleaning up lots of small commits: Squash my last X commits together

Feel free to got through the git tutorial and add your profile to the People section of the website (link to your personal website or email or snail mail address). The qualifying criteria for being an official member of THW is a pull request to the People page.

Data Mining and Machine Learning

2018-02-21T00:00:00+00:00

Data Mining

Mark Kamuda: Link to presentation

Machine Learning

Jim Wilson

Attending

Python & C++ style guides

2018-02-07T00:00:00+00:00

Style Guides for Python & C++

Spring Kick-Off & Intro Python Unit Testing

2018-01-17T00:00:00+00:00

Attending

<++>

Intro Python Unit Testing

Bleeding Edge Stuff

Lightning Talks

<+ person +> : <+ topic +>

Matplotlib Tips & Tricks

2017-12-06T00:00:00+00:00

Matplotlib (Python)

The major plotting package for Python is Matplotlib and is the basis for many other popular plotting packages.

Interesting explanation about Matplotlib’s newest default color scheme, based on color natural perception: SciPy 2015 (more than just why jet sucks, which it does)

Code for today’s talk can be found here.

Attending

<++>

Database Your Data

2017-11-15T00:00:00+00:00

Attending

<++>

Databases: HDF5 & SQLite

Data is created and curated in a number of ways and standardized databases can offer a lot help to researchers, experimental and computational.

HDF5
SQLite

Lightning Talks

<+ person +> : <+ topic +>

Natural Language Processing

2017-11-01T00:00:00+00:00

Attending

<++>

Natural Language Processing

NLP covers both text and speech processing using simple to complex software. This talk will cover…

Bleeding Edge Stuff

Brand spanking new library is brand spanking new

Code examples can be found here.

Lightning Talks

<+ person +> : <+ topic +>

Sphinx Documentation & Python Packaging

2017-10-18T00:00:00+00:00

Meeting Info

Date: Oct. 18, 2017
Time: 11:30 AM - lunch; 12:00 PM - tutorial
Location: 2100 NCSA

Generate API Documentation using Sphinx

presentation

Sphinx Documentation & Python Packaging

Awesome Plots!

2017-10-04T00:00:00+00:00

Attending

<++>

Awesome Plots!

Each member will submit plots to show-off and to get help from members to improve them from good to great!

Upload your plots: Google Doc

The more plots submitted, the more useful it will be to everyone.

Lightning Talks

<+ person +> : <+ topic +>

Introduction to TensorFlow

2017-09-20T00:00:00+00:00

Attending

<++>

Introduction to TensorFlow

TensorFlow is an open-source software library for machine intelligence.

Lightning Talks

<+ person +> : <+ topic +>

Fall 2017 Kick-off

2017-09-06T00:00:00+00:00

Overview

This meeting is to introduce The Hacker Within-Illinois, discuss topics, and schedule speakers for the semester.

Please fill out questionnaire before meeting.

Meeting Info

Date: Sept 6, 2017
Time: lunch - 11:30 AM; meeting - 12:00 pm
- Lunch generously provided by CSE
Location: 2100 NCSA
Topic: Fall 2017 Kick-Off

Attending

<++>

HPC and Parallelization

2016-12-07T00:00:00+00:00

Meeting Info

Date: December 07, 2016
Time: 12:00 PM
Location: 2000 NCSA

Technical: HPC

<+Description Here+>

Lightning Talks:

<+Lightning Talks Here+>

Attendance

<+how many+>

Best Practices

2016-11-30T00:00:00+00:00

Meeting Info

Date: November 30, 2016
Time: 12:00 PM
Location: 2000 NCSA

Technical: Best Practices

Paul Yang presents best practices in scientific computing.

<+Description Here+>

Lightning Talks:

<+Lightning Talks Here+>

Attendance

<+how many+>

Statistical Methods

2016-11-16T00:00:00+00:00

Meeting Info

Date: November 16, 2016
Time: 12:00 PM
Location: 2000 NCSA

Technical: Statistical Methods

Andy Long

<+Description Here+>

Lightning Talks:

<+Lightning Talks Here+>

Attendance

<+how many+>

Testing

2016-11-02T00:00:00+00:00

Meeting Info

Date: November 02, 2016
Time: 12:00 PM
Location: 2000 NCSA

Technical: Testing

Katy Huff

Python

This lesson will bounce around in the following notebooks:

C++

This lesson will bounce around the examples in the Cyclus code suite, which use GoogleTest.

https://github.com/cyclus/cyclus

Lightning Talks:

<+Lightning Talks Here+>

Attendance

<+how many+>

Make, CMake, Docker, & Vagrant - Making it easier to build software

2016-10-19T00:00:00+00:00

Meeting Info

Date: October 19, 2016
Time: 12:00 PM
Location: 2000 NCSA

Technical: Make, CMake, Docker

Greg Hart Alex Lindsay Katy Huff

Make Introduction

Make is software that allows you to carry out a series of tasks in a way that respects the dependencies amongst files or tasks. While it was develovoped for bulding software (i.e. compiling C++ code, linking the files, and placing the executable) It can be used for more then that. The Make manual states “You can use make with any programming language whose compiler can be run with a shell command. Indeed, make is not limited to programs. You can use it to describe any task where some files must be updated automatically from others whenever the others change.” I will cover the basics of Make and give an idea of how powerful and complex it can be.

Basic Makefile

Here is a basic make file:

$cat Makefile
test: test.o anotherTest.o
        gcc -Wall test.o anotherTest.o -o test

test.o: test.c
        gcc -c -Wall test.c

anotherTest.o: anotherTest.c
        gcc -c -Wall anotherTest.c

clean:
        rm -rf *.o test

Now running make will compile both .c files and link them into the executable test:

$make
gcc -c -Wall test.c
gcc -c -Wall anotherTest.c
gcc -Wall test.o anotherTest.o -o test

Here is what is happening. When you run make it looks for a file named Makefile in the current directory and looks at the first target (thing/name on the left of the colon). It then “makes” this target by running the associated command(s) (the indented line(s) following the target) after checking the prerequisites (the things/names after the colon).This checking can lead to other targets being made.

Here specifically the target test depends on test.o and anotherTest.o so make checks for files/targets with these names. It finds the targets test.o and anotherTest.o and then checks their prerequisites test.c and anotherTest.c, but there are no targets for test.c or anotherTest.c so the rules are executed. Make checks if test.o’s time stape is older then test.c’s since test.o doesn’t exist its timestamp is older and the rule is run (the indented lines executed). The same happens for anotherTest.o. With these prerequistites taken care of make goes back to the rule for test and makes it since test is “older” then it’s prerequisites.

Since rules are only executed if the prerequites have changed since the target was made if we run make again nothing will happen:

$make
make: Nothing to be done for `test'.

Make makes incremental builds by only building targets if something changed that effects them. For example if we edit test.c and run make:

$ make
gcc -c -Wall test.c
gcc -Wall test.o anotherTest.o -o test

anotherTest.o isn’t recompiled because the changes to test.c do not affect it. For this simple example this is not a big deal, but for more complicated projects this can save a lot of time.

Make automatically builds the first target in the file. However you can specify the target by listing it after make (i.e. make clean). It is very common to have a makefile (such as this exmaple) with a target executable and a clean target that deletes the executable and all the .o files. Often clean is used if you want to rebuild everything regradless of when files were edited, however the -B option unconditionally makes all targets. If your makefile is not named Makefile use -f file. If your makefile (and everything else) is not in the working directory use -C dir. Make will actually switch to this directory and run then switch back.

Adding variables

In a makefile you can use varibles. The are declered like this:

CXX = g++

and are used like this:

$(CXX)

$cat Makefile
CC = gcc
Warnings = -Wall

test: test.o anotherTest.o
        $(CC) $(Warnings) test.o anotherTest.o -o test

test.o: test.c
        $(CC) -c $(Warnings) test.c

anotherTest.o: anotherTest.c
        $(CC) -c $(Warning) anotherTest.c

clean:
        rm -rf *.o test

With these varables we only have to edit one line if we want to change the compiler or turn off the warnings by commenting out the line that defines Warnings. Alternately we could add a -g to the variable Warnings if we wanted to debug our code.

There are magic variables that are important to know about. The above file could be rewritten as:

$cat Makefile
CC = gcc
Warnings = -Wall

test: test.o anotherTest.o
        $(CC) $(Warnings) $^ -o $@

test.o: test.c
        $(CC) -c $(Warnings) $<

anotherTest.o: anotherTest.c
        $(CC) -c $(Warning) $<

clean:
        rm -rf *.o test

Here $@ becomes the target, $< is the first prerequesite, and $^ is all the prerequisites. This has made the file less readable, but doesn’t offer any benefit in this simiple example.

A More complicated example

Here is a “real” makefile based on one from my research:

CXX = g++
DEBUG = n
USE_OMP = n
CXXFLAGS = -O2 -I/share/apps/include -I./include -I/usr/lib64 -I/usr/lib64/atlas -DUSE_BLAS -DUSE_LAPACK
LIBS = -L/share/apps/lib -l:libtrng4.a -L./lib -lblas /usr/lib64/atlas/liblapack.so
CXXLDFLAGS = -O2

#if DEBUG is set to y add debug flags to CXXFLAGS
ifeq ($(DEBUG),y)
        CXXFLAGS += -DDEBUG
        CXXFLAGS += -g
endif

#if USE_OMP is set to y add libraries for OMP
ifeq ($(USE_OMP),y)
        CXXFLAGS += -fopenmp
        LIBS += -lgomp
endif

#create a variable, CPPS, with all the files in the directory that end in .cpp
CPPS = $(wildcard *.cpp)
#create a variable, OBJS, with the extensions to the sorce files changed from .cpp to .o
OBJS = $(CPPS:.cpp=.o)
#create a list of the file names without extensions
SOURCE = $(foreach file,$(CPPS),$(notdir $(basename $(file))))

#create the executable dependent on all the object files
PottsOMP: $(OBJS)
        $(CXX) $(CXXLDFLAGS) $(OBJS) -o $@ $(LIBS)

#From each .cpp file in the directory create a rule for a .o file of the same name dependent on this .cpp file. There is another (deprecated) way of doing this .cpp.o:
%.o: %.cpp
        $(CXX) $(CXXFLAGS) -c $<

clean:
        rm -f PottsOMP *.exe *.o *~

#This target creates a file, depends.mk, with the depencies for each .o file. This saves one the trouble of trying to keep track of the .h files themselves.
depends : 
	@ rm -f depends.mk
	@ for f in $(SOURCE); do $(CXX) -MM $$f.cpp -MT $$f.o >> depends.mk; done

#include the file created by the above target
include depend.mk

$cat depend.mk
main.o: main.cpp main.h functions.h ran2.h mersenne.h grid.h blas.h lapack.h
functions.o: functions.cpp functions.h ran2.h
ran2.o: ran2.cpp ran2.h
mersenne.o: mersenne.cpp mersenne.h
grid.o: grid.cpp grid.h

Here I am defining a number of variables to set the compiler and compiler flags. Then I create a list of all the .cpp files in this directory and create a corresponding list of .o files. Next is the target for the executable. The next is strange but powerful. Here the % acts as a wildcard and we get a target for very .cpp file in the directory. This way we do not have to write out the rules for each file. It does not matter if the number of files change. We will have a rule for each one. Next I have the standard clean target. Next is a line creating a seperate file with depences and then an include line to include this file.

=================

Other uses for Make

Many people like to mention that Make can be used for things other than compiling code. However it is very hard to find any examples of this. Several people mention using it to manage a website but I couldn’t find any examples of that. This stackover posts talks about using Make for parallel bash scripts.

Mike Bostock says, “Makefiles are machine-readable documentation that make your workflow reproducible”. He suggests using them for downloading data, processing data, and making plots.

CMake

Katy will start with the official CMake tutorial and will then skim two examples of real-world CMake use (Cyclus and HDF5).

Lightning Talks:

Attendance

<+how many+>

Git and GitHub

2016-10-05T00:00:00+00:00

Meeting Info

Date: October 05, 2016
Time: 12:00 PM
Location: 2000 NCSA

Technical: Git and GitHub

Aaron Anderson

<+Description Here+>

Lightning Talks:

<+Lightning Talks Here+>

Attendance

<+how many+>

Python

2016-09-21T00:00:00+00:00

Meeting Info

Date: September 21, 2016
Time: 12:00 PM
Location: 2000 NCSA

Technical: Python

Paul Yang

<+Description Here+>

Lightning Talks:

<+Lightning Talks Here+>

Attendance

<+how many+>

Python Tutorial

2016-09-20T00:00:00+00:00

Overview

I will present on basic usage of python for array manipulation, plotting, text parsing and database manipulation. The tutorial file is available on Github.

Meeting Info

Date: Sep 21, 2016
Time: 12:00 PM
Location: 2000 NCSA
Topic: Python Tutorial

Bourne Again - Intro to BASH and Scripting

2016-09-07T00:00:00+00:00

Meeting Info

Date: September 07, 2016
Time: 12:00 PM
Location: 2000 NCSA

Technical: Bourne-Again: Intro to BASH and Scripting

Andy Long

This session will cover a brief introduction to the Bourne-Again SHell (BASH), covering the following topics:

File system navigation
File manipulation
Pipes
Conditionals
Loops
Functions

Using these techniques, we will walk through an example bash script used to compute the prime factorization of an integer.

Lightning Talks:

<+Lightning Talks Here+>

Attendance

<+how many+>

Fall 2016 Kick-off

2016-08-24T00:00:00+00:00

Overview

This meeting is to introduce The Hacker Within-Illinois, discuss topics, and schedule speakers for the semester.

Please fill out questionnaire before meeting.

Meeting Info

Date: Aug 24, 2016
Time: lunch - 11:30 AM; meeting - 12:00 pm
Location: 2000 NCSA
Topic: Fall 2016 Kick-Off

Attending

<++>

Predicting Earth Surface Temperature with Machine Learning

2016-04-27T00:00:00+00:00

Overview

Machine learning can be applied to many areas and predicting the surface temperature is helpful for many areas of science. We’ll work through an example supplied by Kaggle.

Meeting Info

Date: Apr 27, 2016
Time: 12:00 PM
Location: 2000 NCSA

Using Brain Connectivity and Machine Learning to Understand Skill Learning

2016-03-02T00:00:00+00:00

Overview

Machine learning approaches offer an important advantage over classical approaches to statistical prediction by preventing overfitting, thus creating models that are more generalizable and likely to replicate. In the current presentation, I will introduce a simple to use machine learning package that we created to analyze neuroimaging data, and I demonstrate how we’ve applied it to real data. Acquisition of motor skills is thought to depend highly on interaction between the motor system and subcortical structures. In our demonstration, we used this machine learning package to examine how functional brain network connectivity in the subcortical and motor networks predict initial performance on a complex motor task, as well as training-induced improvements following 20 hours of training. By contrasting the predictive success of different types of feature sets, we explore how the connectivity both within and between these networks contain differential information for predicting individual performance and learning rate. We also interrogate sets of features that are most reliably included in the models with high predictive accuracy, and discuss what these highly successful feature sets imply for learning and performance of complex motor tasks.

Meeting Info

Date: Feb 3, 2016
Time: 12:00 PM
Location: 2000 NCSA
Topic: Using Brain Connectivity and Machine Learning to Understand Skill Learning

##Bio:

Aki Nikolaidis investigates how brain networks, metabolism, and structure contribute to individual differences in executive function, intelligence, and learning rates. His work has also focused on understanding how cognitive training drives brain plasticity and improvements in cognition. More broadly, he is interested in using cutting edge statistical methodology to understand how patterns of large scale brain plasticity drive forms of learning that generalize across multiple contexts.

High Performance Computing with Accelerators

2016-02-17T00:00:00+00:00

Overview

Senior Research Scientist Dr. Volodymyr Kindratenko will give an overview of his research with High Performance Computing with Accelerators. If members have a need or interest in trying accelerators, then he will provide access to THW members.

Meeting Info

Date: Feb 3, 2016
Time: 12:00 PM
Location: 2000 NCSA
Topic: High Performance Computing with Accelerators

Jump Start Julia with Path Integral Monte Carlo (PIMC)

2016-02-03T00:00:00+00:00

Overview

Interpretive languages are easy to use but slow. Compiled languages are fast but hard to develop. Julia seeks to deliver the best of both worlds as an interpretive language that is as fast as Fortran. Julia is also a modern language that is designed with cloud computing, distributed computing and data mining in mind. Participants will experience the ease and speed of Julia by implementing the path integral Monte Carlo algorithm to solve a quantum harmonic oscillator and compare speeds to a python equivalent. Jupyter notebooks for this tutorial are available at julia notebook and python notebook.

A short summary of PIMC can be found here.

A short feature demonstration of julia can be found here.

Meeting Info

Date: Feb 3, 2016
Time: 12:00 PM
Location: 2000 NCSA
Topic: Julia vs. Python in PIMC

##Notes:

Julia installation scripts for Fedora 23 and Ubuntu 14.04. Syntax highlighting in vim can be nicely integrated through vundle.

Building a Website with Github

2016-01-20T00:00:00+00:00

Meeting Info

Date: Jan 20, 2016
Time: 11:30 AM - lunch; 12:00 PM - tutorial
Location: 2000 NCSA

Building a Website with Github

Creating a website is a great way to have a professional presence dedicated to you and your work. As a computational scientist/engineer, utilizing Github’s pages free webshosting allows you to get up and runnig quickly, it helps you become more comfortable with git, connects you to your contributions to open-source projects, and your stand apart from your other colleagues.

This tutorial will walk you through the steps of initializing the website (or cloning a template) and add your first content.

2 ways to build website

Github Pages intro
“Good hackers copy, great artists steal!” - Free Jekyll Themes - Github user website template - UIUC alum

Attendance

<++>

Spring '16 planning

2015-12-02T00:00:00+00:00

Meeting Info

Date: Dec 2, 2015
Time: 12:00 PM
Location: 1040 NCSA

Spring ‘16 Planning

There has been a lot of interest in THW at Illinois but we need to continue to provide the resources students want to help them excel!

Questions to answer:

What has and hasn’t worked?
- Technical talks (e.g. git, bash, make, NAMD/VMD)
- Reserach talks
- Meeting time
What topics should be covered?
Is there interested in long-term learning?
- Cousera “Data Science” specialization
- “Effective Computing in Physics”
What kind of professional development would be helpful?
- resume, cover letter, presentation templates
- industry speakers
Is there interest in outreach?
- teaching coding to middle or high school kids?
Day-long or half-day bootcamps?

Attendance

<++>

Make Introduction

2015-11-11T00:00:00+00:00

Meeting Info

Date: Nov 11, 2015
Time: 12:00 PM
Location: 1040 NCSA

Technical: “Make Introduction”

Gregory R. Hart

You can use make with any programming language whose compiler can be run with a shell command. Indeed, make is not limited to programs. You can use it to describe any task where some files must be updated automatically from others whenever the others change.

I will cover the basics of Make and give an idea of how powerful and complex it can be.

Lightning Research Talks:

###”Towards uncertainty quantification of groundwater models with structural error” Tianfang Xu (CSE Fellow)

Physically-based numerical models of groundwater flow are powerful quantitative tools to manage scarce groundwater resources. Inherent uncertainties associated with model structure and parameter lead to both random and systematic errors even in the output of a calibrated model. In this talk, I will present a complementary data-driven modeling framework to quantify the predictive uncertainty of groundwater models. The framework constructs error models based on machine learning techniques to correct for model structural error. The postprocessor and fully Bayesian implementations of the framework are illustrated using synthetic and real-world case studies.

###”Introduction to prostate cancer diagnosis using quantitative phase imaging and machine learning” Tan Nguyen

I will give a brief introduction to the problem of prostate cancer diagnosis using the standard-of-care staining method in contrast with our new method using the high throughput none-invasived Quantitative Phase Imaging (QPI) and machine learning. A machine learning algorithm is implemented to learn textural behaviors of prostate samples imaged under QPI and produce labeled maps of different regions for testing biopsies (e.g. gland, stroma, lumen etc.). From these maps, morphological and textural features are calculated to produce diagnosis results of the testing samples.

Attendance

<++>

BASH basics--script your workflow

2015-10-28T00:00:00+00:00

Meeting Info

Date: Oct 28, 2015
Time: 12:00 PM
Location: 1040 NCSA

Technical: “BASH basics: script your workflow”

Aaron T. Anderson

BASH is one of the most common command line programming environments in Linux/Unix. Some basic scripting commands will be covered to help automate common, repetitive tasks in the command line.

Lightning Research Talks:

###”Vortex-induced Vibrations of a Sprung Cylinder with Nonlinear Internal Elements” Antoine Blanchard

Structures immersed in a flow may experience vibrations due to their interaction with the surrounding fluid, which over time can cause fatigue or failure. We use direct numerical simulations to gain insight into the underlying dynamics of such interaction, and design novel devices that can mitigate undesired vibrations of the structure.

Attendance

<++>

Hands-on Tutorial of NAMD and VMD

2015-10-14T00:00:00+00:00

Meeting Info

Date: Oct 14, 2015
Time: 12:00 PM
Location: 1040 NCSA
Topic: Hands-on Tutorial of NAMD and VMD

Technical: “Hands-on Tutorial of NAMD and VMD”

Maxim Belkin

Molecular dynamics simulations help us understand the origins of molecular behavior at the nanometer and femtosecond scales. In this session we will prepare and run molecular dynamics simulations of DNA in a solid-state (graphene) nanopore using VMD and NAMD.

NAMD

VMD

Lightning Research Talk: “Vortex-induced Vibrations of a Sprung Cylinder with Nonlinear Internal Elements”

Antoine Blanchard

Attendance

<++>

Molecular Modeling with NAMD

2015-09-30T00:00:00+00:00

Meeting Info

Date: Sept 30, 2015
Time: 12:00 PM
Location: 1040 NCSA
Topic: Molecular Modeling with NAMD

Technical: “Molecular Modeling with NAMD”

Maxim Belkin

Computational modeling of biological systems becomes an increasingly popular method to study molecular behavior at the nanoscale. NAMD is a program developed at UIUC for performing molecular dynamics (MD) simulations on a wide variety of computer systems ranging from laptops to supercomputers. We will discuss the main concepts of MD simulations and go over basic steps for setting up and running simulations with NAMD.

Lightning Research Talk: “Atomic-scale dynamics of glass-forming metallic liquids”

Abhishek Jaiswal

Bulk metallic glasses are interesting materials to explore the universality of glass transition. They are characterized by complex interactions, yet have simpler structure lacking internal degrees of freedom. In this talk, I will present my current work that utilizes MD simulations and machine learning techniques to characterize and visualize the underlying nature of dynamics in the liquid state of a model ternary Cu-Zr-Al system. I will discuss on the universal nature of dynamics between metallic glasses and other glass-forming systems including molecular, covalent bonded glasses etc.

Attendance

Jaehyung Yu
Jinsheng Wang
Zhikun Cai
Nathan Walter
Jaiswal Abhishek
Antoine Blanchard
Tianfang Xu
Yubo Yang
Greg Hart
Bryce Thurston
Tan Nguyen
Ahmad Raeisi Najafi
Masoud Safdari
Aaron Anderson
Mohamed Mohamed

Git Workflow

2015-09-16T00:00:00+00:00

Meeting Info

Date: Sept 16, 2015
Time: 12:00 PM
Location: 1030 NCSA
Topic: Git Workflow

Technical: “Git Workflow”

Aaron T. Anderson

Git is version control software best understood in by using it in the context of a workflow for collaborating on a code base. Attendees will get a better understanding of git by looking at the theoretical workflow, a practical example, and how to recover from a conflict.

Last meeting’s fixed example: Git tutorial – “add new post”

External Resources (in order of complexity):

Try git live in your browser - great tutorial!
Git Reference - favorite reference
Git Cheatsheet
Simple workflow
Collaborating workflow

THW Community resources:

THW-Berkeley Intro
- Intro to Git I
- Intro to Git II
THW-Berkeley Fall 2015
- Git Intro
- Git Advanced

Lightning Research Talk: “Error Catastrophe — HIV like melting ice”

Gregory R. Hart

HIV is a rapidly evolving virus which makes it difficult to treat or cure. By drawing comparisons between between the viral quasispecies and statistical physics we can calculate the “thermodynamic” properties of the virus, finding a “phase transition”. This phase transition is reveals novel drug treats and new treatment strategies.

Prez Tip #2

Git + LaTeX

LaTeX is a popular typesetting system in academia because it’s focus on the simple construction of mathematical formulas. Since LaTeX is like any other coding language, it is well suited to be tracked by git.

List of (suggested) branches:

master – most publishable version
advisor – track suggestions for edits
sections – branch for each, to make tracking changes easier
- abstract
- introduction
- methods
- results
- discussion
- conclusion

When someone suggests edits, create a new branch (or update advisor branch) and save two versions of the paper, e.g. paper.tex and paper_advisor.tex, then run latexdiff:

$ latexdiff paper.tex paper_advisor.tex > paper_advisor_highlighted.tex
$ pdflatex paper_advisor_highlighted.tex

The pdflatex command will output a marked-up version of the paper (removed words = crossed out red; added words = blue and underlined squiggle).

More tips and suggestions can be found on the StackOverflow question titled “Git + LaTeX workflow”.

Attendance

Greg Hart
Nicholas Laracuente
Tan Nguyen
Trang Vu
Antoine Blanchard
Tiangfang Xu
Aki Nikolaidis
Subhadeep De
Sikandar Mashayak
Stuti Shrivastava
Noel Naughton
Yubo Yang

Prostate cancer diagnosis using QPI and machine learning

2015-09-09T00:00:00+00:00

Attending

<++>

New Hat

Should be a good thing to show.

Bleeding Edge Stuff

Brand spanking new library is brand spanking new

Code examples can be found here.

Lightning Talks

<+ person +> : <+ topic +>

Git Version Control Overview

2015-08-26T00:00:00+00:00

Overview

Version control software helps researchers track and document code development and is essential for combining the work of groups developing on the same code base. A very popular and open source version control software is Git and will be introduced through the use of the online code repository Github. Participants will have the opportunity to contribute to an open source project using git and Github.

Tutorial

Meeting Info

Date: Aug 26, 2015
Time: 12:00 PM
Location: 1030 NCSA
Topic: Git Version Control Overview

Lightning Research Talk

The development of state-of-the-art simulation model that self-consistently couple the electronic and phonon transport is essential in creating a cycle that will push designs to have lower carbon footprints and in creating environmentally conscious electronics that minimize waste. We provide a quick primer on the physics of nanoelectronic devices as well as some of the key challenges faced.

Notes:

THW-Berkley: Intro to Git I

THW-Berkley: Intro to Git II

Prez Tip #1

A helpful way to view the progression of your git repository is with a handy shortcut:

$ git config --global alias.la "log --all --graph --date=short --format=format:\"%C(yellow)%h%Creset %cd%Cred%d %Cblue%cn%Creset %s\""

Then just type:

$ git la

Source: Professor Matt West’s git quick-ref

Attendance

David Hannasch
Tan Nguyen
Yubo Yang
Greg Hart
Sam Kaufman
Abhishek Jaiswal
Nathan Walter
Andy Loftus
Aki Nikolaidis
Tanvien Talukdar
John Capozzo
Patricia Watson
Brain McGuigen
Chloe Ma
Tianfang Xu
Sikandar Mashayak
Terry Fleury
Noel Naughton
Michael Katolik
Jim Basney
Rylan Dmello
David S. Ancalle
Antoine Blanchard
Stuti Shrivastava

Kick-Off Meeting

2015-08-19T00:00:00+00:00

Overview

This meeting is to introduce The Hacker Within, Illinois chapter and discuss what students want out of the group.

Meeting Info

Date: Aug 19, 2015
Time: 11:30 AM
Location: 3100 NCSA
Topic: Kick-Off Meeting

Questions to answer

What is THW-IL?
What can be expected?
What are topics of interest?
How can both undergraduate and graduate students benefit?
How to promote the group?

[meeting_url]: