Building AI-Driven Apps Using Semantic Kernel.pptx
Bash and regular expressions
1. BASH Scripting aanndd RREEGGEEXX bbaassiiccss
FFrreeddLLUUGG
HHooww ttoo ddeemmyyssttiiffyy BBAASSHH aanndd RREEGGEEXX
Peter Larsen
September 27, 2014 Sr. Solutions Architect, Red Hat
2. AAggeennddaa
● Introduction to BASH
● Basics – with exercises
● A little more advanced
● Introduction to REGEX
● Understanding the Shellshock bug
Sep 27, 2014 2
3. BBAASSHH IInnttrroodduuccttiioonn
● Shell – what is it and who cares?
● Current directory, umask
● Exit codes
● Functions
● Built in commands
● Environment Variables
● Traps, Options and Aliases
● sh, ksh, csh and the rest of the family
● How does it work?
● Command Expansion
● Command Execution
Sep 27, 2014 3
4. CCoommmmaanndd EExxppaannssiioonn
● Variable assignments and redirections are saved for later
processing
● Words are expanded. In the result, the first word is taken
to be the command and the rest are arguments
● Redirections are executed
● Text after = are expanded/substituded
Sep 27, 2014 4
5. CCoommmmaanndd EExxeeccuuttiioonn
● If no / in command shell searches for command
● If function, calls function
● If shell built-in, execute
● Search $PATH
● To execute
● Create subshell
● Run command in subshell
● If not found, return exit code 127
Sep 27, 2014 5
6. BBAASSHH -- BBaassiiccss
● Hello World
● Variables
● Passing arguments to a script
● Arrays
● Basic Operations
● Basic String Operations
● Decision Making
● Loops
● Shell Functions
Sep 27, 2014 6
7. HHeelllloo WWoorrlldd
● Anything after # is ignored
● Blank lines are ignored
● Start all bash scripts with #!/bin/bash
● To execute a bash script, set execution bit or run:
bash[scriptname]
● Always provide path to script or place it in $HOME/bin
Sep 27, 2014 7
8. HHeelllloo WWoorrlldd --EExxaammppllee
● Create a script that writes „Hello, World!“ on the screen
● The „print“ command in bash is called echo – it's an
internal bash command.
#!/bin/bash
echo "Hello, World!"
Sep 27, 2014 8
9. VVaarriiaabblleess
● Variables are created when assigned
● Syntax: VAR=VALUE
● Note: No spaces before/after the =
● Case sensitive – one word, can contain _ but not other special characters
● Read variables by adding $ in front of it or use ${var}
● Use to escape special characters like $
● Preserve white-spaces with “ “
● Assign results of commands to variables with ` (back-ticks) or $(command)
Sep 27, 2014 9
10. VVaarriiaabblleess –– EExxaammpplleess
PRICE_PER_APPLE=5
MyFirstLetters=ABC
greeting='Hello world!'
PRICE_PER_APPLE=5
echo "The price of an Apple today is: $HK $PRICE_PER_APPLE"
MyFirstLetters=ABC
echo "The first 10 letters in the alphabet are: ${MyFirstLetters}DEFGHIJ"
greeting='Hello world!'
echo $greeting now with spaces: "$greeting"
FILELIST=`ls`
FileWithTimeStamp=/tmp/my-dir/file_$(/bin/date +%Y-%m-%d).txt
Sep 27, 2014 10
11. VVaarriiaabblleess -- EExxeerrcciissee
#!/bin/bash
# Change this code
BIRTHDATE=None
Presents=None
BIRTHDAY=None
# Testing code - do not change it
if [ "$BIRTHDATE" == "Aug 11 1967" ] ; then
● Create 3 variables in the sample code:
● String (BIRTHDATE) – contain the text “11 1967”
echo BIRTHDATE is correct, it is $BIRTHDATE
● Integer (PRESENTS) else
– contain the number 10
echo "BIRTHDATE is incorrect - please retry"
● Complex (BIRTHDAY) fi
– contain the weekday of $BIRTHDATE
(Friday)
if [ $Presents == 10 ] ; then
echo I have received $Presents presents
else
● Hint: use the 'date' command to get the weekday from a
date
date -d “$date1” +%A
echo "Presents is incorrect - please retry"
fi
if [ "$BIRTHDAY" == "Friday" ] ; then
echo I was born on a $BIRTHDAY
else
echo "BIRTHDAY is incorrect - please retry"
fi
Sep 27, 2014 11
12. VVaarriiaabblleess -- SSoolluuttiioonn
#!/bin/bash
# Change this code
BIRTHDATE="● BIRTHDATE="Aug 11 1967"
Sep 27 2014"
Presents=10
BIRTHDAY=$(Presents=date -d "$BIRTHDATE" +%A)
# Testing ● code - do 10
not change it
if [ "$BIRTHDATE" == "Sep 27 2014" ] ; then
● echo BIRTHDAY=$(BIRTHDATE is correct, date it -d is "$$BIRTHDATE" BIRTHDATE
+%A)
else
echo "BIRTHDATE is incorrect - please retry"
fi
if [ $Presents == 10 ] ; then
echo I have received $Presents presents
else
echo "Presents is incorrect - please retry"
fi
if [ "$BIRTHDAY" == "Friday" ] ; then
echo I was born on a $BIRTHDAY
else
echo "BIRTHDAY is incorrect - please retry"
Sep 27, 2014 12
fi
[fredlug@fredlug class]$ bash ./var-solution.sh
BIRTHDATE is correct, it is Aug 11 1967
I have received 10 presents
I was born on a Friday
13. Passing aarrgguummeennttss ttoo aa ssccrriipptt
● Arguments are passed to a script when it's run
● Arguments are given after the command line with spaces between them
● Refer to arguments inside the script with:
● $1 first argument
● $2 second argument
● Etc.
● $0 is the script name
● $# number of arguments
● $@ all parameters space delimited
Sep 27, 2014 13
15. AArrrraayyss
● Several values in the same variable name
● Created with space separated values in ( )
● Total array values: ${#arrayname[@]}
● Use ${array[index]} to refer to values
● Note index numbers start at 0 (not 1).
Sep 27, 2014 15
17. AArrrraayy -- EExxeerrcciissee
● Create a bash script
● Define array NAMES with 3 entries: John, Eric and Jessica
#!/bin/bash
NAMES=( John Eric Jessica )
● Define array NUMBERS with 3 entries: 1, 2, 3
● Define variable NumberOfNames containing the number of
names in the NAMES array using $# special variable
# write your code here
NUMBERS=(1 2 3)
NumberOfNames=${#NAMES[@]}
second_name=${NAMES[1]}
echo NumberofNames is: $NumberOfNames
echo second_name is: $second_name
● Define variable second_name that contains the second name in
the NAMES array
● Print the content of NumberOfNames and second_name
[fredlug@fredlug class]$ bash ./array.sh
NumberofNames is: 3
second_name is: Eric
Sep 27, 2014 17
18. BBaassiicc OOppeerraattiioonnss
● Use $((expression))
● Addition: a + b
● Subtraction: a – b
● Multiplication: a * b
● Division: a / b
● Modulo: a % b (integer remainder of a divided with b)
● Exponentitation: a ** b (a to the power of b)
Sep 27, 2014 18
19. BBaassiicc OOppeerraattiioonnss -- EExxeerrcciissee
● Given
● COST_PINEAPPLE=50
● COST_BANANA=4
● COST_WATERMELON=23
● COST_BASKET=1
#!/bin/bash
COST_PINEAPPLE=50
COST_BANANA=4
COST_WATERMELON=23
COST_BASKET=1
TOTAL=$(( $COST_BASKET +
( $COST_PINEAPPLE * 1 )
+ ( $COST_BANANA * 2 )
+ ( $COST_WATERMELON * 3 ) ))
echo Total is: $TOTAL
● Calculate TOTAL of a fruit basket containing 1 pinapple, 2
bananas and 3 watermelons
● Print the content of TOTAL
$ bash ./operations.sh
Total is: 128
Sep 27, 2014 19
20. BBaassiicc SSttrriinngg OOppeerraattiioonnss
● STRING=”this is a string”
● String length: ${#STRING} →16
● Numerical position of character:
expr index $STRING “a” →9
● Substring: ${STRING:$POS:$LEN)
● POS=1, LEN=3 →his
● ${STRING:12} →ring # from pos and to the end of var
● Substring replacement: ${STRING[@]/string/text} →this is a text
● Substring replace ALL: ${STRING[@]//is/xx}→thxx xx a string
● Delete all occurrences: ${STRING[@]// a /}→this is string
● Replace first occurrence: ${STRING[@]/#this/that/}
● Replace last occurrence: ${STRING[@]/%string/text}
Sep 27, 2014 20
21. SSttrriinnggss -- EExxeerrcciissee
#!/bin/bash
BUFFET="Life is like a snowball.
The important thing is finding wet snow and
a really long hill."
● Given BUFFET="Life is like a snowball. The important thing is
finding wet snow and a really long hill."
● Create ISAY variable with the following changes:
ISAY="$BUFFET"
ISAY=${ISAY[@]/snow/foot}
echo First: $ISAY
ISAY=${ISAY[@]/snow/}
echo Second: $ISAY
● First occurence of 'snow' with 'foot'
$ bash ./string2.sh
First: Life Delete is like a football. The important Second: ● Life is like second a football. occurence The important of snow
thing is finding wet snow and a really long hill.
thing is finding wet and a really long hill.
Third: Life is like a football. The important thing is getting wet and a really long hill.
Fourth: Life is like a football. The important thing is getting wet
● Replace 'finding' with 'getting'
● Delete all characters following 'wet'
● Print ISAY
ISAY=${ISAY[@]/finding/getting}
echo Third: $ISAY
POS=`expr index "$ISAY" "w"`
ISAY=${ISAY:0:POS+3}
echo Fourth: $ISAY
Sep 27, 2014 21
22. DDeecciissiioonn MMaakkiinngg
● If [ expression ]; then
code the true part
else
code the false part
fi
● Else can be replace with elif if followed by another if
● Case $variable in
“condition1”)
command ...
;;
“condition2”)
command ...
;;
esac
mycase=1
case $mycase in
1) echo "You selected bash";;
2) echo "You selected perl";;
3) echo "You selected python";;
4) echo "You selected c++";;
5) exit
esac
Sep 27, 2014 22
24. DDeecciissiioonn mmaakkiinngg -- EExxeerrcciissee
● Change variables to make expressions true
#!/bin/bash
# change these variables
NUMBER=10
APPLES=12
KING=GEORGE
# modify above variables
# to make all decisions below TRUE
if [ $NUMBER -gt 15 ] ; then
echo 1
fi
if [ $NUMBER -eq $APPLES ] ; then
echo 2
fi
if [[ ($APPLES -eq 12) || ($KING = "LUIS") ]] ; then
echo 3
fi
if [[ $(($NUMBER + $APPLES)) -le 32 ]] ; then
Sep 27, 2014 24
echo 4
fi
NUMBER=16
APPLES=16
KING=LUIS
25. LLooooppss
● “for” loop
for arg in [list]
do
command(s) ....
done
● “while” loop
while [ condition ]
do
command(s) ...
done
● “until” loop
until [ condition ]
do
command(s) ...
done
● “break” - skip iteration
● “continue” - do next loop
now
Sep 27, 2014 25
26. LLoooopp EExxaammpplleess
# Prints out 0,1,2,3,4
COUNT=0
while [ $COUNT -ge 0 ]; do
# loop on array member
NAMES=(Joe Jenny Sara Tony)
for N in ${NAMES[@]} ; do
echo Value of COUNT is: $COUNT
COUNT=$((COUNT+1))
if [ $COUNT -ge 5 ] ; then
echo My name is $N
done
break
# loop on command fi
output results
for f in $( ls *.sh /etc/localtime ) ; do
done
# Prints out only odd numbers - 1,3,5,7,9
COUNT=0
while [ $COUNT -lt 10 ]; do
echo "File is: $f"
done
COUNT=4
while [ $COUNT -gt 0 ]; do
echo Value of count is: $COUNT
COUNT=$(($COUNT - 1))
done
COUNT=1
until [ $COUNT -gt 5 ]; do
echo Value of count is: $COUNT
COUNT=$(($COUNT + 1))
done
COUNT=$((COUNT+1))
# Check if COUNT is even
if [ $(($COUNT % 2)) = 0 ] ; then
continue
fi
echo $COUNT
Sep 27, 2014 26
done
27. LLoooopp EExxeerrcciissee
#!/bin/bash
NUMBERS=(951 402 984 651 360 69 408 319 601 485
980 507 725 547 544 615 83 165 141 501 263)
for num in ${NUMBERS[@]}
do
● NUMBERS=(951 402 984 651 360 69 408 319 601 485
980 507 725 547 544 615 83 165 141 501 263)
● Print all even numbers in order of array
● Do not print anything after 547
if [ $num == 547 ]; then
break
fi
MOD=$(( $num % 2 ))
if [ $MOD == 0 ]; then
echo $num
fi
done
$ bash ./loops.sh
402
984
360
408
980
Sep 27, 2014 27
28. SShheellll FFuunnccttiioonnss
function function_B {
● Sub-routine that implements echo Function set B.
of commands and
operations.
}
function function_A {
echo $1
● Can take parameters
}
function adder {
● Useful for repeated tasks
}
# FUNCTION CALLS
# Pass parameter to function A
function_A "Function A." # Function A.
function_B # Function B.
# Pass two parameters to function adder
adder 12 56 # 68
● function_name {
command ....
}
echo $(($1 + $2))
Sep 27, 2014 28
29. FFuunnccttiioonnss -- EExxeerrssiizzee
#!/bin/bash
function ENGLISH_CALC {
● Write a function ENGLISH_CALC which process the
following:
NUM1=$1 ; OPTXT=$2 ; NUM2=$3
case $OPTXT in
plus) OP='+' ;;
minus) OP='-' ;;
times) OP='*' ;;
*)
echo Bad operator $OPTXT
;;
esac
echo $NUM1 "$OP" $NUM2 = $(($NUM1 $OP $NUM2))
● ENGLISH_CALC 3 plus 5
● ENGLISH_CALC 5 minus 1
● ENGLISH_CALC 4 times 6
● The function prints the results as 3 + 5 = 8, 5 – 1 = 4 etc.
}
ENGLISH_CALC 3 plus 5
ENGLISH_CALC 5 minus 1
ENGLISH_CALC 4 times 6
Sep 27, 2014 29
30. BBAASSHH –– AAddvvaanncceedd
● Special Variables
● Bash trap command
● File testing
● There's a lot more features – this is not comprehensive
● $man bash is your friend
Sep 27, 2014 30
31. SSppeecciiaall VVaarriiaabblleess
* $* = “$1 $2 $3 ......” $ Process ID of shell
@ $@ “$1” “$2” “$3” ..... ! Process ID of most recent
background process
# Number of parameters 0 Name of shell or program being
executed
? Exit status _ Aboslute path of shell or command
- Current option flags (shopt)
Sep 27, 2014 31
32. BBaasshh ttrraapp ccoommmmaanndd
● “trap” executes a script automatically when a signal is
received
● $ trap program sigspec
● List all signals with “trap -l”
● Great for catching a HUP or INT to clean up temporary
files etc before exiting
Sep 27, 2014 32
33. FFiillee tteessttiinngg
● Used as a condition to set actions based on file attributes
● Exists, readable, writable etc.
● File1 older/newer than File2
● Commonly used in if statements [ ] [[ ]] etc.
Sep 27, 2014 33
34. FFiillee TTeessttiinngg ooppttiioonnss
● -f regular file exists
● -d directory exists
● -h symbolic link exists
● -r file is readable
● -w file is writable
● file1 -nt file2: file1 newer
than file2
● file1 -ot file3: file1 older
than file2
● file1 -ef file2: file1 and file2
refers to same inode
Sep 27, 2014 34
35. RReegguullaarr EExxpprreessssiioonnss
● Characters/Strings
● Character Classes and Bracket Expressions
● Anchoring
● Backslash and special expressions
● Repetition
● Concatenation, Alternation, Precedence
Sep 27, 2014 35
36. DDeemmoo ffiillee
● Create a file resolv.conf with the following contents
● Make sure grep is aliased to:
grep –color=auto
; generated by /sbin/dhclient-script ^$[](){}-?*.+:_
search brq.com mylab.brq.com lab.eng.brq.com world.com
nameserver 12.14.255.7
nameserver 14.14.255.6
38. Character Classes aanndd BBrraacckkeett EExxpprreessssiioonnss
● [ ] is a set of characters that matches. A string matches if it
matches any of the characters in the set. A ^ inside the [ ]
means do not patch
● Predefined sets like [[:alnum:]] [[:digit:]] exists to make
writing easier
● Decimal point (.) matches any single character
Sep 27, 2014 38
40. AAnncchhoorriinngg
● Locks the search pattern to a specific position
● ^ beginning of line
● $ end of line
Sep 27, 2014 40
41. Backslash aanndd ssppeecciiaall eexxpprreessssiioonnss
● Backslashes can prefix special
functions
● < = Start of word
● > = End of word
● b = beginning of word
● B = not b
● w = word
● W = not word
Sep 27, 2014 41
42. RReeppeettiittiioonn
● * repeats 0 or more times
● + repeats 1 or more times
● ? repeats 0 or 1 time
● {5} repeats 5 times
● {2,3} repeats 2 or 3 times
Sep 27, 2014 42
43. Concatenation, AAlltteerrnnaattiioonn,, PPrreecceeddeennccee
● Concatenation: sequence of characters (literal/special)
● Alternation: Separate different patterns with |
● Precedence: Parentheses, Repetition, Concatenation,
Alternation
● Use ( ) to group things together for later reference
Sep 27, 2014 43
45. #shellshock –– tthhee ffaammoouuss BBAASSHH bbuugg
● So what is it? CVE-2014-6271
● Try this at your command prompt:
x='() { :;}; echo vulnerable' bash -c "echo test"
● Does it print vulnerable? If so, you need to update your
BASH right away.
● $ rpm -q bash
Should report version 4.2.45-5.4 – if not “yum update” now.
Sep 27, 2014 45
46. ##sshheellllsshhoocckk -- hhooww
● A little known “hack” allows functions to be treated as
variables
● $ function foo { echo "hi mom"; }
$ export -f foo
$ bash -c 'foo' # Spawn nested shell, call 'foo'
hi mom
● Great Blog:
http://lcamtuf.blogspot.com/2014/09/quick-notes-about-bash-bug-Sep 27, 2014 46
47. ##sshheellllsshhoocckk –– hhooww ccoonnttiinnuueedd
● $ foo='() { echo "hi mom"; }' bash -c 'foo'
hi mom
● Let's break it down
foo='() { echo “hi mom”;}'
“magic property” () {
is executed before the command is run
execute bash running foo
● Since env variables are used by httpd, dhcpd and other daemons, it
potentially allows them to run code by simply setting a value in a
variable.
Sep 27, 2014 47