Fastlane tutorial for production use

 Fastlane is a set of open sources tools(written in ruby) for automating various aspects like generating build for iOS, Android, running unit tests etc.

Here is how I did fastlane set up & integration to the existing CI-Jenkins. We have used fastlane for the CD part of CI/CD pipeline.

Commands

• these are few commands I have used. a lot are there.

• fastlane init 

  • creates FastFile, AppFile

  • Firsttime for a new acc, failed due to lack of certs, provisioning profiles. this will take apple id, password.

• fastlane cert - to create a distribution profile using above acc

• fastlane sigh - to create/get provisioning profile for the above to appstore. not for adhoc

• fastlane beta - init will create a betalane, now running this command will archieve & export .ipa & push to TestFlight.

• fastlane pilot upload - to push to store can use

• fastlane match - to use existing certificates. its pushes/pulls certs required for iOS from a git repo and do the job.

Setting up fastlane

• Install: brew install fastlane

• in source code folder go to each platform run: 

  • fastlane init

    • this will ask what we want to do- select 2, Automate push to TestFlight

    • next, choose the right target - App

    • give a different appname whichc is not in your account

  • run fastlane beta

    • But, we need to get certs from apple developer portal & need to be used by everyone. for that, we need to use match (a fastlane tool)

    • run fastlane match init

    • it will ask for a priavte git repo. created and gave.

    • it will create a matchfile

    • it will ask for apple username, password.

    • Based on the current project id, it will create the required certificates, provisioning profile

    • for test flight, ran: fastlane match appstore.

    • the above will create the cert if not there.

    • in our case, we have exceeded max no.of Dist certs so, it throwed an error. so, now setting up a git repo with certs and profiles that fastlane and all of us can use.

    • to use the existing certs, there is no direct tool. we need to use irb(interactive ruby tool or a ruby script to fetch the certs), encrypt, decrypt them.

    • wrote a script beforePlacingManualCerts.rb for fetching the certs & placing it in required folder and to upload.

    • to get id, need to use spaceship cli command. Ref: https://medium.com/@jonathancardoso/using-fastlane-match-with-existing-certificates-without-revoking-them-a325be69dac6

    • Then, fastlane match appstore —readonly. (this is giving error as of now.)

    • Add run script phase to remove unused architectures: 

    • create an App specific password 

    • setup ssh for match git repo

    • Gym-for building iOS.

    • 1st time, it may ask for accessing login key chain, enter password and always allow. this steps need to be checked on automating.

    • React-native-config - no space, commans, quotes etc.

    • https://stackoverflow.com/a/40661027/4065202 for libRealmJS not found etc. adding a new Config need to do these. "$(BUILD_DIR)/Release$(EFFECTIVE_PLATFORM_NAME)"

    • need to input password once firstime building and always allow.
      

Fastlane iOS implementation to push to AppStore is completed. here are the things that were done.

AppStore:

• created a gmail account(your apple account), registered as developer in apple developer portal.

Fastlane:

• created a lane called beta to push/run the set of commands/actions to push to TestFlight.

• First, will check for required certificates, provisioning profile

  • This will be carried out by Match.

  • Created a match file.

  • For easy sharing and use of certificated across the team, created a private git repo.

  • All the Certificates & profiles will be here & All will be encrypted.

  • Access to the git repo is restricted. not all of you will have access. it will be provide on a need basis for safety.

  • Here, as we need to use existing certificates, no direct fastlane tool is there to do this job. So, wrote a Ruby script for automating the manual job of running few terminal commands to get the existing certificates and place them in to the git repo with encryption. t

  • Checked in this automation as well with name UseExistingCertsProfiles_Fastlane.rb.

• Match will fetch certs and install it in local keychain. It will ensure everytime certificates are upto date with certificates in apple portal.
  

• This will make sure that all required certificates for the app are there.

• Automations for store builds:

  • For majority of the required automations, i have used react-native-config plugin.

  • All the required config variables are moved to config file called .env & .env.production. Config variables are

    • DEFAULT_ENV in Config.js

    • DEV_FEATURES in Constants.js

    • NEWRELICE_AGENT_KEY - appdelegate.m

    • REALM_SCHEMA_VERSION - offlineServices.js

    • For firebase GoogleServices-info.plist i have created a Build Phase: Copy Firebase GoogleService-Info.plist - which runs a shell script to place the file in build folder based on Configuration selected.

    • For removing unused architectures required for JWPlayer, created another Build Phase: Remove Unused architectures - (JW iOS SDK.framework signing issue resolution) - which runs a shell script to identify and remove unused architectures.

    • For the above purpose to not to be effected by regular dev builds, created a new Scheme: app.production & Configuration: Store.

    • Added pre-build step to set the ENVFILE required for react-native-config

    • Manual tasks

      • I left two of the items to be manual as automation will complicate than simplifying. they are:

        • Version: 1.8.3 etc. This follow semantic versioning: major.minor.patch

        • To let fastlane increment correctly, we need to pass the above bump type. I can’t default it to any. So, left it.

        • Realm Schema

          • This is required in a JS file OfllineServices.js

          • Need to use Git hook to update the version value in .env file. but that comes under keyword expansion. 

          • Git suggests not to do it as it may lead to merge conflicts.

          • Doing that using Git is also an overkill. So, left it.

        • We need to make sure to update the Version number, Schema version correctly in the release/master branch.

• Jenkins
  

  • Installed required libraries Fastlane, bundler etc

  • created a job in Jenkins (in a separate folder) to run the above fastlane script.

  • Added Lang environment variable in Jenkins which is required for fastlane.

Note:

• Only when running for the first time, we need to provide login key chain password and click on Always allow. This is required fot accessing the certificate for signing.
  

Ofcourse, resolved many certificate issue, building/linker issues (it iOS).
Slack integration & iOS

• Android:

  • Supply - upload_to_play_store

  • should be in json format

  • app-fastlane-service-acc - API service acc name - android management user

  • key id: xxxxxxxxxxxxxxxxxx

  • newrelic key change - done

  • Google services json - done

  • Slack -

    • Fastlane-mobile, web hook: https://hooks.slack.com/services/xxxxx

    • android done. add error

    • iOS - done. add error

  • Job with progress

1. created a google account and got invited by playstore admin

2. With PlayStore owner credentials created a service account for pushing the builds to play store.

3. added fastlane to android

4. Config and constants are taken care in iOS, so not required to be done here.

5. for new relic using react-native-congfig, added step to read from .env file

6. placed the google-services.json file in android/app/src/release folder. Firebase plugin will pick from the dev/release folders respectively.

7. added slack notifications by using weblinks.

1. For this, created an app in slack and get it approved by admin (Shiva)

2. created a weblink to #mobile-pipeline

3. using that in Fastfile to send messages

Ref:

• https://medium.com/react-native-training/fastlane-for-react-native-ios-android-app-devops-8ca85bee614e
• https://rollout.io/blog/react-native-developer-series-deploy-ios-and-android-pipelines-with-fastlane-and-jenkins/
• https://docs.fastlane.tools/best-practices/continuous-integration/jenkins/
• https://www.macadamian.com/learn/automating-ios-certificate-and-profile-syncing-with-fastlane/
• https://medium.com/@hohey670/ios-multiple-configurations-with-fastlane-8cbaff7d2fa8
• https://medium.com/@jonathancardoso/using-fastlane-match-with-existing-certificates-without-revoking-them-a325be69dac6
• https://medium.com/ios-development-tips-and-tricks/slack-fastlane-%EF%B8%8F-about-one-of-continuous-integration-puzzles-6ca03576003e

UseExistingCertsProfiles_Fastlane

##############------SMALL AUTOMATION-------##########

# instead of doing the job of uploading existing certificates via irb (interactive Ruby) manually, this helps in automating

# to use it, give your git_url(where you are storing your cert, profiles for match), match_password below and keep the folder stucture as mentioned below

#-------------------FOLDER-STRUCTURE------------------

# to use this automation follow the folder structure mentioned here:

# Fastlane match keeps certificates in cert directory, provisioning profiles in profiles directory

# for different envs like development, distribution certificates will be kept in respective folders under certs folder. i.e.

# - certs/development for dev; certs/distribution for dist;

# - profiles/appstore for appstore; profiles/adhoc for adhoc etc.

# keep your certs and profiles then same way inside fastlane folder (can be any but keeping in fastlane folder for ease)

# so fodler structure is: fastlane/certs/distribution, fastlane/certs/development. fastlane/profiles/appstore etc.

#--------------------RUNNING--------------------------

# ruby UseExistingCertsProfiles_Fastlane.rb

#

require 'match'

git_url = 'https://github.xxxxxxx.xxxx/xxx/xxxxxxx.git'

shallow_clone = false

ENV["MATCH_PASSWORD"]='xxxxxxxx'

branch = 'master'

storage = Match::Storage.for_mode('git', { git_url: git_url, shallow_clone: shallow_clone, git_branch: branch, clone_branch_directly: false})

storage.download

encryption = Match::Encryption.for_storage_mode('git', { git_url: git_url, working_directory: storage.working_directory})

encryption.decrypt_files

# copy file

puts __dir__+"/certs"

FileUtils.copy_entry(__dir__+"/certs", storage.working_directory+"/certs")

FileUtils.copy_entry(__dir__+"/profiles", storage.working_directory+"/profiles")

puts(storage.working_directory)

encryption.encrypt_files

files_to_commit = Dir[File.join(storage.working_directory, "**", "*.{cer,p12,mobileprovision}")]

storage.save_changes!(files_to_commit: files_to_commit)

###************NOTE***********######

# this can be completly automated using Ruby.

# First check if certs & profiles folders exist and has subfolder required or not.

# accept git_url, password, may be branch as well in case multiple apps as commandline arguments

# rest works fine.

# Because of lack of time, not doing it now, other can take it up & contribut to opensource

Fuchsia

Google is working on launching a new OS for anything that could fit under Google’s tech umbrella starting from smart phones, smart speakers to laptops and IoT devices.

It’s an attempt to get entire Google’s ecosphere under one operating system.

 

Few points I came to know while reading about this:

• Android was built before iPhone was release. Android was built for the use in Digital cameras later due to iPhone launch it was repurposed.
• Flutter SDK (with Dart programming language) can be used for building apps for new OS as well along with iOS, Android and desktop.
• Its following micro kernel approach.
• Its built-on top of Zircon micro kernel. (read this & above link if you want to recollect some names from Operating System subject in Bachelor’s)
• Microsoft has tried building another operating system called Sigularity.
   

Ref: digitaltrends, techcrunch

Statistics basics - 1

if("you are an absolute beginner in data science") 
{
        One simple line which summarizes the various works in data industry is "Data Science, Data   Analytics, Machine learning etc are different sides of the same multi-sided polygon".
        If you are aiming to step into any of these a strong understanding of statistics will help to pioneer. Statistics expertise is mandatory for data scientist, machine learning experts where as basic is OK for other roles.
       Role of statistics in understanding, analyzing, automated decision taking is inseparable. So, Set your mind to read, learn, understand and apply statistics to see great success in data industry.
}
.....................
      To construct a giant, sustainable building, its foundation should be strong. similarly, strong understanding of statistics is necessary for anybody in data industry. so, let strengthen the basics of statistics.
will touch up on:

1. Central Tendency
2. variance, standard deviation
3. Distributions, Normal distribution
4. Confidence intervals, degrees of freedom

1. Central Tendency:
          A measure of central tendency is a single value that attempts to describe a set of data by identifying the central position within that set of data.They are also classed as summary statistics. The mean (often called the average) is most likely the measure of central tendency that you are most familiar with, but there are others, such as the median and the mode.
a. Mean(Average)
        The mean is equal to the sum of all the values in the data set divided by the number of values in the data set. So, if we have n values in a data set and they have values x1, x2, ..., xn, the sample mean, usually denoted by  (pronounced x bar), is:

 

Ex: Mean for a given set can be computed as follows: 

• Mean can be calculated for both continuous and discrete data as well.
• It is the only center tendency measure which considers each data input.
• It suffers with Outliers.

b. Median

       Median is the middle value in a sorted data set. For a data set with even number no.of elements, average of both the values will be the median

x = [23, 40, 6, 74, 38, 1, 70]

sorted_x = [1, 6, 23, 38, 40, 70, 74]

Median is 38.

• unlike mean, it doesn't suffer much because of  outliers

c. Mode
         Most common value in the data set. Mode is most useful when you need to understand clustering or number of ‘hits’. For example, a retailer may want to understand the mode of sizes purchased so that he can set stocking labels optimally. Say, store A has a mode of ‘small’ while store B has a mode of ‘XXL’.

Based on type of variables, we need to choose the right central tendency measure. Cheat sheet for it is,

 

 Read about types of variables here.

Now, we have read about measures of central tendency. i.e. What is the central value of the data set. It talks about center values, doesn't talk about how data was spread. To understand that, we need to consider,

1. Variance(σ^2)
2. Standard Deviation (σ)

Unlike range and quartiles, the variance combines all the values in a data set to produce a measure of spread.
 1. Variance

variance is a measure of how spread out a data set is.Variance is the average of the squared differences from the mean. 

Variance (S²) = average squared deviation of values from mean

Difference is squared to avoid cancelling of values above mean and below mean.

As the inputs are squared, units are not same for Variance and the data. 

Standard Deviation, is the Square root of the variance.

2. Standard Deviation:(σ)
        It measures spread around the mean. Because of its close links with the mean, standard deviation can be greatly affected if the mean gives a poor measure of central tendency. Understanding Standard deviation is very important to understand the data spread. its the one referred in six sigma technique.

       The data set with the smaller standard deviation has a narrower spread of measurements around the mean and therefore usually has comparatively fewer high or low values. An item selected at random from a data set whose standard deviation is low has a better chance of being close to the mean than an item from a data set whose standard deviation is higher.

Generally, the more widely spread the values are, the larger the standard deviation is.

 

       Although standard deviation is less susceptible to extreme values than the range, standard deviation is still more sensitive than the semi-quartile range. If the possibility of high values (outliers) presents itself, then the standard deviation should be supplemented by the semi-quartile range.

If  = mean, σ = standard deviation and x = a value in the data set, then

• about 68% of the data lie in the interval:  - σ < x <  + σ.
• about 95% of the data lie in the interval:  - 2σ < x < + 2σ.
• about 99% of the data lie in the interval:  - 3σ < x <  + 3σ.

Next Posts:
3. Distributions and Normal distribution:


4. Confidence intervals, degrees of freedom

Flutter and Dart

Google has released a new Hybrid framework for developing mobile apps named FLUTTER. It’s based on their new programming language called DART.

It’s official website: https://flutter.io/ . As it is hybrid don’t think as like cordova but non-cordova i.e. no-bridging (external) SDK.

Its inspired by FB’s react native but it takes hybrid apps to another level in terms of performance and development speed. Here is a detailed analysis, don’t miss it. Article focuses on the anatomy of Dart and Flutter.

A point of note is that google claiming it will be a big hit as it is not just for mobile but it revolutionizes web development.  Dart 2 is out now and It’s interesting…

here is an informational quora question:  https://www.quora.com/Is-Flutter-likely-to-replace-Java-for-Android-app-development

fix bootstrap errors with angular 4

Uninstall bootstrap@4.0.0-beta.3:
npm uninstall bootstrap --save
Then installed bootstrap@4.0.0-beta.2:
npm install bootstrap@4.0.0-beta.2 --save

OR 

upgrade angular cli from 1.6.3 to 1.7.0-rc.0. but this has some problems with fonts

but issues coming up with both, click events aren't working for hamburger in mobile view.

so, i finally felt better to go with ng-bootstrap or the mdbootstrap (material design). material isn't yet in a production mode. 

here are the steps to install ng-bootstrap.

1.  npm install --save @ng-bootstrap/ng-bootstrap
2. add the following to algular-cli.json - styles:[], so it would be styles:["../node_modules/bootstrap/dist/css/bootstrap.min.css", "styles.css"]
3. start using the angular bootstrap components looking at: https://ng-bootstrap.github.io/#/getting-started 

angular material design: https://material.angular.io/guide/getting-started

switch between DEV, QA and PROD easily in angular 4/ionic 3

While building any apps its important for a developer to switch between the environments.
below is a simple way i felt for switching environment in angular 2/4 or ionic 2/3.

its easy with a provider and associative arrays. below is an example:

create a provider called environments and use the below:

_envs = {
dev:{
baseURL:"https://somedev.exaple.net/api/mobile",
    config:{
    serviceRefName:"/demo1",
    serviceRefname2:"/demo2",
    serviceRefname3:"/demo3",
    //...
    //...
    }
}
    qa:{
    baseURL:"https://someqa.exaple.net/api/mobile",
    config:{
    serviceRefName:"/demo1",
    serviceRefname2:"/demo2",
    serviceRefname3:"/demo3",
    //...
    //...
    }
    },
    prod:{
    baseURL:"https://someprod.exaple.net/api/mobile",
    config:{
    serviceRefName:"/demo1",
    serviceRefname2:"/demo2",
    serviceRefname3:"/demo3",
    //...
    //...
    }
    }
 };

 _env={};
 _selectedEnv = "qa";

 //add this line in constructor
 this._env = this._envs[this._selectedEnv];

 //add this method to the environment provider
 getURL(configUrl){
    console.log("configUrl Sec : "+configUrl);
    //console.log("this._env="+JSON.stringify(this._env));
    console.log("URL="+this._env["baseURL"]+this._env["config"][configUrl]);
    return this._env["baseURL"]+this._env["config"][configUrl];
  }

In components, where you want to call a service,

• add environment provider dependency
• use this:  this.environment.getURL("YourSericeName")

for ex: this.environment.getURL("serviceRefname3")

whenever you need to change, simple change the _selectEnv in you Environment provider.
 For ex:
 _selectedEnv = "prod";

It will take the service names from prod obj of _env. this way with a very simple step, you can switch environment in angular.

Note: It has nothing much specific to angular. this logic can be used in any programming language with respective object constructs.

MODS 17

OCT 5

• A11Y - for accessibility not disability 
• tota11y.com,  a11y.org, koa11y.tool
• Scott Davis company - Thirsty Head
• Future is all micro services, currently its all monolithic
• Kubernetes - IBM's ., container orchestration
• Swift for web
• REPL
• Chris Latner is a genius
• Protocol oriented
• LLVM?
• CodableProtocol - Swift 4 JSON type
• TaskStorage - delete, save etc 4 props
• Kitura
• Open API 2.0 Spec - swagger
• React Native
• DogFooding
• Hot Reloading
• Configuring instant updates?
• Expo - Dev tool
• expo kit for managing custom needs
• CRNA
• Testing:
• Jest - Snapshot Testing
• Enzyme
• StoryBook
• Interactive Dev & Testing
• Loki - Visual Regression Testing
• Monkey Runner tests
• Build & Deploy:
• fastlane
• Tools:
• Lottie - Animations
• Crashlytics, Sentry
• Expo Snack
• Detox - replacement for appium/selenium
• appitools - Visual automated testing
• Open STF - for certain devices testing
• Electrode native - convert to react
• Building for Next Billion  Users:
• about PWA's
• Performance in google chrome set to slow 3G 
• In slow 3g expected performance:
• First meaningful paint - 1.6s
• Time to interactive - 2.5s
• Page Ready - 3.6s
• React with GraphQL & Apollo:
• Redux - State Management
• Graph QL is server side, which decouples Server endpoints and client
• GraphQL
• Only one end point will be there 
• Client has to query details it needs and GraphQL will call corresponding service/db and executes a query and returns queried data.
• Its kind of agreement between client and server so, client need not bother about service side changes. it will always return the details queried by the client and in the same format as the query.
• Query - Read, Mutation - Write, Subscription - Push
• cycle: Query -> Resolvers (Functions to calculate how to fetch data) -> Connectors
• Apollo:
• Client side for GraphQL mostly. it replaces the need for Relay.

OCT 6

• Data - J.Paul Getty's quote
• Deep leaning:
• Yousha Bengio book on deep learning
• Super long sequence pattern recognition
• Alpha GO -by google
• Over fitting in Neural networks
• BFF (Backend for frontend) with swift:
• Micro services
• Be part of swift evolution - Chris Latner
• Swift on linux works on Ubuntu 14.0 onwards, linux one server platform and android platform as well
• Foundation & Dispatch are in progress for apple & Linux platforms
• Web service frame works based on swift are  - Kitura, Perfect, Vapour
• Swift is best for web interms of performace and storage when compared with Node, Java
• Citizen Data Scientist:
• epilipsy?
• Lyfas
• ibm.biz/heartmotion - ibm's project
• PCA - principle component analysis
• Data science experience in Bluemix
• Deep Learning by (methods)
• blundering - experimentation
• transfer learning - japan cucumber example
• Simulation
• Swift Stack developer is a upcoming profile
• Robust introduction AI:
• Recurrent neural networks
• Tensor layer
• Appless environment:
• Conversational UI
• Intents
• Entity
• Action
• Cloud functions (Google) & Lamda (AWS)
• Siri - apple, Google Assistant - Google, Amazon - Alexa
• console.actions.google.com - for cloud functions
• Amazon echo, echo dot,
• Earcon instead of icon
• Siri kit by apple for ConvoUI
• Chat bubbles - Response
• Mood Analysis - Urgency etc
• Seamless UI - b/w devices etc
• RxJS: @ladyleet
• IxJS - Matt Podwyski is working on
• Sync .of, .from  -synchronus functions in Rx
• .share?
• SpeechRecognition - mozilla/api/web_search_api
• .switchmap?
• .merge?
• Native Script:
• from teleric like react
• snippets plugin for easy coding of native script in atom