Hacker Within

One year on: a review of THW, lessons learned and next steps

2019-04-15T00:00:00+00:00

when: Monday 15 April, 1-2pm.
where: Murray Learning Centre UG05
what: Review of the first year of THW, lessons learned and next steps”

Presenter

Matthew Brett, founder of THW at UoB, will be discussing “One year on: a review of THW, lessons learned and next steps”.

Applying THW lessons to a research project

2019-03-18T00:00:00+00:00

<+ Flaviu Gostin +>

<+ Flaviu Gostin is a researcher. He uses X-ray diffraction to characterise corrosion products generated by titanium alloys for medical implants. +>

<+ Applying The Hacker Within lessons to a research project +>

<+ In the scientific community, problems may arise when different labs cannot reproduce the work of each other. As research nowadays involves more and more computation, it is essential to create data analysis workflows that can be reproduced by others, i.e. computational reproducibility. Previous THW talks introduced a range of tools from the software development world, which can be used to create reproducible workflows to analyse research data. +>

<+ Before coming to the talk +>

<+ For this talk the presenter prepared a repository on GitHub containing a workflow to analyse some example data presented in his recent paper. One of the most important test of reproducibility is to have your workflow being reproduced by another person on a different machine. Before this talk, please try to reproduce the presenter’s workflow on your machine: just follow the instructions in his GitHub repo here.

Hacking copyright : An Anecdotal History from Saint Columba to Richard Stallman

2019-01-21T00:00:00+00:00

when: Monday 21 January, 1-2pm.
where: Murray Learning Centre UG05
what: Hacking copyright

Presenter

Chen Zhu from Birmingham Law School will be discussing “Hacking Copyright: An Anecdotal History from Saint Columba to Richard Stallman”

Presentation:

Compiling software on Linux - the good, the bad and the ugly

2018-11-19T00:00:00+00:00

when: Monday 19 November, 1-2pm.
where: Murray Learning Centre UG05
what: Compiling software on Linux - the good, the bad and the ugly

Presenter

James Carpenter and Andrew Edmondson from Advanced Research Computing

Reproducible research

2018-10-15T00:00:00+00:00

when: Monday 15 October, 1-2.30pm.
where: Murray Learning Centre UG10
what: Reproducible research

Presenter

Simon Branford, from the Research Software Group in Advanced Research Computing. You can find the slides here

A brief introduction to data.table

2018-09-17T00:00:00+00:00

when: Monday 17 September, 1-2.30pm.
where: Murray Learning Centre UG07
what: A brief introduction to data.table

A brief introduction to data.table” or “If Thor was an R Programmer, His Hammer Would Be data.table”

Abstract

In this 20-minutes talk, I will present basic examples of the use of data.table which is an R package that provides an enhanced version of data.frames. I will also share some of my experiences with data.table and ggplot2 for time-series analyses. Slides and exercise are available here: https://github.com/mxochicale/thw-r-datatable

Presenter

Miguel P Xochicale (twitter: @_mxochicale)

Final Year Doctoral Researcher doing Human-Robot Interaction, Nonlinear Dynamics, Deep Learning and Open Science.

Testing and continuous integration

2018-08-20T00:00:00+00:00

when: Monday 20 July, 1-2.30pm.
where: Muirhead 112
what: Testing and continuous integration

Stream of talk

Try this link

Getting set up

Make sure you have these two things on your computer:

A Python distribution. If you haven’t got one already, and particularly if you are on Windows, I recommend the Anaconda distribution installer. Choose the Python 3.6 version, not the Python 2.7 version. You do not need to install VSCode, when that option comes up.
Git version control software. I suggest you choose the option to use the Nano editor by default, instead of Vim (I use Vim, but if you are not used to it, it’s a terrible choice for getting started as quickly as possible).

Improving as a programmer

Does an academic need to be a good programmer?
What is a good programmer?
- Teach yourself programming in 10 years.
- Winning a programming competition is not what you need.

Preparing to use Github

Go to https://github.com
If you haven’t got an account, make one now. Use your .bham.ac.uk address, it gives you access to some education features (if you ask for them).
Navigate to https://github.com/matthew-brett/play-poker
Click on the “Fork” button just below the top right of the screen. Wait for a bit. You should now be looking at your own fork of my play-poker repository.
Select the URL of your new forked repository from the URL bar, and copy it. The URL will be of form https://github.com/your-user-name/play-poker.
Open the terminal (Cmd-spacebar “Terminal” on Mac, Start key “git bash” on Windows)
Change directory to the Desktop:
```
cd Desktop
```
Clone your new repository with the following at the terminal:
```
git clone https://github.com/your-user-name/play-poker
```
where your-user-name is your Github username.
Change directory to the repository directory.
```
cd play-poker
```

Ready for poker

If you’re on Windows - press the “Start key” then type “Spyder” (no quotes). Press return. The Spyder editor should start. Click on File, select Open, navigate to the Desktop play-poker folder, and select poker.py. Click the green run (play) icon to run the file.
On Mac, in Terminal.app, check you are in the Desktop/play-poker directory with:
```
pwd
```
Now run your poker.py script with:
```
python poker.py
```
You should see the printout of “Running tests”, and “Finished”. Congratulations, you are all set for some test-driven development.

Getting more interactive

When we are writing tests, we often want to experiment with code, and look at variables. The best tool to do this is the IPython interactive Python prompt.

Windows: if you are using Spyder, you already have this - it’s the bottom right pane in your Spyder window arrangement.
Mac: first check you have IPython installed by typing:
```
python -m IPython --version
```
If this gives an error, type:
```
pip install --user ipython
```
Now type:
```
python -m IPython
```

Now you will have a prompt of form In [1]: . Type 2 + 3 and press return. This should execute and return 5.

Start with a failing test

This is the key element of test first developement - write a failing test, then fix it.

Edit poker.py. In Windows, you probably already have Spyder open. On Mac, start a new tab in Terminal, with the Shell menu, New Tab option. In this new tab, type nano poker.py.
We are going to start by generating a new desk of cards. To do this, we imagine a make_new_deck function, that returns a fresh unshuffled deck of cards. We start by imagining what that should return. It should probably be something that has 52 cards in it, at least. Imagining further, replace test_new_deck like this
```
def test_new_deck():
    # We will test the `make_new_deck` function
    deck = make_new_deck()
    assert len(deck) == 52
```
Note the four space indentation for the stuff inside the function.

Now run the file - Windws: click Run / Play in Spyder; Mac: switch back to the original tab, running IPython and run run poker.py.

It fails, because we haven’t written make_new_deck yet.
Put a new function called make_new_deck into poker.py. It returns 52 None values.
```
def make_new_deck():
    return [None] * 52
```
Run the tests again. They pass.

Working out the design by testing

Now we have 52 “cards”, but the cards are Python’s special None value. We have to decide what a card should be.

Let’s say one card is a string, of form “7 of Spades” or “Jack of Hearts”.

Let’s also say that a new deck will be not-shuffled, so the first card will be the 2 of Hearts. Expand your test function like this:

def test_new_deck():
    # We will test the `make_new_deck` function
    deck = make_new_deck()
    assert len(deck) == 52
    first_card = deck[0]
    assert first_card == "2 of Hearts"

Run the file. It fails because the first card is currently None.

Experimenting as you go

Now we will write a proper function to generate the new deck of cards.

To do this, we would like to try various things at the interactive IPython prompt.

Try typing this at the IPython prompt:

'2 3 4 5 6 7 8 9 10 Jack Queen King Ace'.split()

You should see a length 13 list with the card values.

Try this:

'Hearts Diamonds Clubs Spades'.split()

This should be a four element list of suit names.

We can put these at the top of our poker.py file, like this:

VALUES = '2 3 4 5 6 7 8 9 10 Jack Queen King Ace'.split()
SUITS = 'Hearts Diamonds Clubs Spades'.split()

Run the file from IPython with:

run poker.py

Now check the variables look right by showing their contents. At the IPython prompt:

VALUES

and

SUITS

Let’s try making the first card in IPython.

VALUES[0] + ' of ' + SUITS[0]

To make the full deck of cards, we cycle through all the suits, then all the values, and combine the value with the suit, to make 52 cards:

def make_new_deck():
    cards = []
    for suit in SUITS:
        for value in VALUES:
            card = value + ' of ' + suit
            cards.append(card)
    return cards

Run the tests. Do they pass?

Extend the tests

We just did a lot of code, and our tests only test the first card. We should add some more tests to check our make_new_deck function is doing the right thing.

Add these lines to your test_new_deck function. Be careful to keep the same indentation as the lines above:

    assert deck[12] == "Ace of Hearts"
    assert deck[13] == "2 of Diamonds"
    assert deck[26] == "2 of Clubs"
    assert deck[51] == "Ace of Spades"

Making a poker hand

To make a poker hand, we shuffle the cards, and then give out five cards from the shuffled deck. What would a shuffled deck look like?

First we write a failing test.

A shuffled deck will virtually never have the same order as a fresh deck. Let’s put that in as a test, expecting it to fail one time in 52 x 51 x 50 x 49 etc, and that is basically never.

def test_shuffled():
    ordered_deck = make_new_deck()
    to_shuffle = make_new_deck()
    shuffled_deck = shuffled(to_shuffle)
    assert shuffled_deck != ordered_deck

You’ll also need to add this function call to your if __name__ block:

if __name__ == "__main__":
    print("Running tests")
    test_new_deck()
    test_shuffled()
    print("Finished")

Run the file. It fails. We are going to fix it.

First consider this really bad shuffle function:

def shuffled(deck):
    # A really bad shuffle function
    re_ordered = deck.copy()
    # Reverse the order
    re_ordered.reverse()
    return re_ordered

That would always return a deck with the ‘Ace of Spades’ first and the ‘2 of Hearts’ last. But our test would still pass. Put that function into poker.py.

Run the tests - they pass. Oops. We need to test more.

shuffled needs to do more than return something different from an ordered deck. It needs to return something different each time.

We add some new lines to test_shuffled that will do 1000 shuffles, and makes sure each one is different. We take the opportunity to fold in the check that the deck is not ordered.

def test_shuffled():
    decks = []
    # Put an ordered deck into the list of decks
    decks.append(make_new_deck())
    for i in range(1000):
        deck = make_new_deck()
        shuffled_deck = shuffled(deck)
        # Compare this deck to all previous ones.  Do any match?
        assert shuffled_deck not in decks
        decks.append(shuffled_deck)

Run the tests. They should fail.

How to make this work correctly? Let’s experiment in IPython:

import random

Use tab completion to explore the random module, by typing random. (notice appended period) and then pressing tab. We find random.shuffle. Here is a not-shuffled deck:

deck = make_new_deck()
# First 10 cards
deck[:10]

Now shuffle:

random.shuffle(deck)
# New order shows in first 10 cards
deck[:10]

We will use random.shuffle for our shuffled function.

Type import random at the top of poker.py. Replace the bad version of shuffled with this version:

def shuffled(deck):
    # A proper shuffle function
    re_ordered = deck.copy()
    random.shuffle(re_ordered)
    return re_ordered

Run the tests. They should pass.

A better way to run the tests

pytest is a Python package that makes it easier to write tests and run them.

So far we’ve been running the tests by running our Python module poker.py. We had to add a call to our test functions in the if __name__ block.

Pytest finds tests in .py files automatically, and then runs the tests.

If you installed Anaconda, you have Pytest installed already.

If not, open a terminal window, and type:

python -m pytest --version

If you get No module named pytest, then:

pip install --user pytest

Now check you can run the tests with:

python -m pytest poker.py

This is an efficient way to find tests and run them.

Commit your changes

Open Git (go back to git bash in Windows; open a new Terminal tab on Mac).

First check that Git knows your name and email. Type:

git config user.name

It should show your name. If it doesn’t show anything, type:

git config user.name "Your Name"

Now check:

git config user.email

If that shows nothing:

git config user.email "my.name@bham.ac.uk"

Now you are ready to prepare and make a new commit. Type:

git status

You should see that poker.py has been modified.

Put it in Git’s staging area ready for the next commit:

git add poker.py

Use git status to show that the changes in poker.py are now in the “Changes to be committed”.

Do the commit with:

git commit

Your text editor should open for you to type a commit message. Save the message to finish the commit.

Continuous integration

Notice the .travis.yml file in the repository. This is a configuration file for the free Travis-CI continuous integration service.

Now we ask Travis-CI to run our tests each time we do a new commit to the repository.

Go to the https://www.travis-ci.com front page. Click on the green button “Sign up with Github”. Click on the green button at the bottom of the page: “Authorize travis-pro”. Select the play-poker repository.

All done. The next time you send a commit to the play-poker repository, Travis-CI will run the tests for you.

Let’s do that:

git push origin master --set-upstream

Now go to the https://travis-ci.com site. You should see that Travis-CI is running or has run your tests for you.

Git and version control

2018-07-16T00:00:00+00:00

when: Monday 16 July, 1-2pm.
where: Nuffield G13
what: Git and version control

Mark Slater talked about version control using the Git package including an overview of what git does under the hood. You can find the slides here

Language Gallery

2018-06-25T00:00:00+00:00

when: Monday 25th June, 1-2pm.
where: Murray Learning Centre UG07
what: Language Gallery

We’ve been talking about different languages used for academic computing, and one topic that has come up several times is - which language should I use?

In this meeting, we talk about these 4 popular languages, each of which are more or less essential in their own niche:

Python - Warrick Ball. Get Warrick’s slides.
R - Murray Hoggett. Get example code and data files.
Javascript - David Madden. Get David’s slides.
Matlab - Dietmar Heinke.

There was a 10 minute talk on each, in which someone who uses the language said something about what the language is good for, why it’s good, and when it’s not so good.

Good Software Practices

2018-05-21T00:00:00+00:00

The third meeting of The Hacker Within at the University of Birmingham was:

when: Monday 21st May, 1-2.30pm.
where: Room UG07, Murray Learning Centre
what: Good Software Practices

Talk by Andrew (Ed) Edmondson.

Please see Ed’s slides.

The Hacker Toolkit, and What to Learn and What to Teach

2018-04-16T00:00:00+00:00

The second meeting of The Hacker Within at the University of Birmingham was:

when: Monday 16th April, 1-2.30pm.
where: Room 118 Muirhead
what: The Hacker Toolkit, and What to Learn and What to Teach.

You don’t need to book, please, just turn up.

There will be two parts to this, our second meeting:

Talk: The Hacker Toolkit. The basic tools of a Hacker’s working life; how to choose your tools; how to learn new ones.
Discussion: What would you like to learn? What would you like to teach?

Here are the:

We recorded the various topics that people wanted to hear about on the whiteboard: .

Look at the Github repository for the source material to generate the slides, including copyright information.

Introduction, and what to Learn and Teach for Spring 2018

2018-03-19T00:00:00+00:00

The first meeting of The Hacker Within at the University of Birmingham will be:

when: Monday 19th March, 1-2pm.
where: Room 118 Muirhead
what: Introduction to the Hacker Within; discussion suggestions and voting about what to learn and what to teach.

You don’t need to book, please, just turn up.

Here are the talk slides.

Look at the Github repository for the source material to generate the slides, including copyright information.

After the talk, I mentioned our Berkeley course on reproducible science. If you want to see what the students managed to do in their final projects, have a look at the course Github organization for the students project repositories, beginning with project-. For example project-alpha.