Nans' blogs

Software development Interview questions

Fri, 27 Jan 2023 18:35:00 GMT

⭐

What is testing in software development ?

⭐⭐

What is dependency injection ? What does it enable ?
What is a micro-service architecture ? What is it used for ?

⭐⭐⭐

What is Dependency Inversion Principle (DIP) ?
Describe an example of clean architecture.

Face à la complexité de la crise environnementale, il faut faciliter les prises de décision

Mon, 10 Oct 2022 09:35:00 GMT

Le climat se dérègle, ce n’est plus un débat. On a aussi des problèmes de disponibilité des ressources fossiles, de dérèglement de la biodiversité, de disponibilité de l’eau … Ça commence à faire beaucoup d’infos à digérer.

Une fois qu’on a choisi d’agir pour les causes environnementales, par où commencer ? Comment peut-on prioriser nos actions ?

Sans prétendre répondre complètement à ces questions, je partage ici un morceau de cheminement.

Je vais principalement prendre des exemples liés au monde de l’agro-alimentaire, parce que c’est le domaine que je connais le mieux.

On améliore ce qu’on mesure

Pour vérifier qu’on fait bien quelque chose, en général, on prend des mesures, et ensuite on s’adapte. C’est comme ça que fonctionne un régulateur de vitesse, il mesure la vitesse à un instant T, puis il demande au moteur d’adapter sa cadence en fonction de ce qu’il mesure.

Schéma du principe de mesure et de rétro-action d’un régulateur de vitesse

Grosso modo, ça ressemble à ça.

En principe, on peut essayer de contrôler n’importe quelle grandeur avec ce genre de système. C’est comme ça qu’on peut faire un métro automatique, ou même qu’on commande les pales d’un hélicoptère.

Après cette petite aparté technologique, revenons au sujet qui nous intéresse. Est-ce qu’on peut mesurer un impact environnemental ?

Évidemment, tout le monde connait l’empreinte carbone. Les plus habitués de cette notion connaissent même les subtilités du “CO2 équivalent”, avec la prise en compte des différents gaz à effet de serre.

On peut même comprendre ce que ça représente avec des simulateurs proposés par l’ADEME comme Mon convertisseur CO2.

Mais si on ne mesure que l’empreinte carbone, on oublie tout le reste ! Est-ce qu’il vaut mieux limiter son empreinte carbone mais aggraver son empreinte eau ?

Empreinte Carbone de produits alimentaires

Empreinte eau de produits alimentaires

Par exemple, lorsqu’on consomme une noix, on a un bilan en CO2 extrêmement faible (c’est une culture qui stocke du carbone), mais on a une empreinte eau très forte.

On a besoin d’une grandeur qui nous permettrait d’éviter les transferts d’impact.

Comment éviter les transferts d’impact ?

La commission Européenne a demandé à des scientifiques de se mettre d’accord sur la question, sous le projet Product Environnemental Footprint, _PEF pour les intimes. On a commencé par déterminer quels sont les indicateurs à mesurer. La Commission Européenne a tranché pour 16 indicateurs. Sans tous les lister, on y retrouve le changement climatique, l’épuisement des ressources en eau, l’épuisement des ressources énergétiques, l’usage des terres, l’eutrophisation terrestre/eau douce/marine …

Le PEF propose ensuite de regrouper ces grandeurs sous un “Single Score”, exprimé en points. Il est obtenu à la suite d’une normalisation et d’une pondération.

Normaliser ça signifie mettre toutes les grandeurs sous une même unité (on ne compare pas du CO2 et des M3 d’eau !).

Pondérer c’est donner plus ou moins d’importance aux différents impacts. Pour le PEF, la pondération prend à la fois en compte la robustesse relative de chacun de ces indicateurs et les enjeux environnementaux.

Single Score pour 1kg de Burritos.

Aperçu du détail des données

Le PEF a des limites malheureusement. Il manque un indicateur spécifique à la biodiversité, il manque aussi un indicateur pour les pollutions au micro-plastique … Mais c’est au moins une méthodologie qui fait consensus chez les scientifiques.

Donc comment on évite les transferts d’impact ? On se dote d’un score le plus global possible, et on ajuste notre régulateur de vitesse de telle sorte à ce qu’il diminue le score.

Comment se servir du Single Score ?

Pour obtenir ce score, on mène une Analyse du Cycle de Vie (ACV). Ce terme désigne une méthode d’évaluation qui permet d’analyser tous les maillons de la chaîne de valeur d’un produit ou service.

Et on parle vraiment de tous les maillons ! Ça part de l’extraction de la matière première, jusqu’à la fin de vie du produit avec le traitement des déchets. Quand on fait l’ACV d’une tomate, on part depuis les minéraux qui constituent le tracteur jusqu’au compost du particulier. On compte aussi le cas ou la tomate est jetée à la poubelle, on compte aussi le cas où la tomate a été mise de côté car “hors calibre” à la ferme.

Extrait de l’ACV du Burritos, de l’usine jusqu’au supermarché

Une fois qu’on a modélisé un produit existant, on peut commencer à imaginer des scénarios d’éco-conception. Si on reprend le schéma du régulateur de vitesse, on va donc envoyer un signal de rétro-action.

On peut faire une première tentative en essayant de chercher à s’approvisionner de façon locale : on supprime tous les transports (oui c’est utopique, mais faisons l’exercice).

Voici le résultat :

Comparaison d’impact Burritos vs Burritos sans aucun transport

On se rend compte que même dans un scénario de réduction maximale des transports, on ne réduit pas de façon significative les impacts.

On identifie clairement que le boeuf haché est un “point chaud” de notre ACV. On a même ce qui ressemble au principe de Pareto : 20% de la matière qui compose mon burritos cause 80% des impacts environnementaux !

Essayons de modéliser un burritos au poulet :

Comparaison d’impact Burritos boeuf vs poulet

Là on a une réduction d’impact bien plus satisfaisante !

Pourquoi cette méthodologie n’est-elle pas plus connue que ça ?

La bonne nouvelle, c’est que l’ACV va bientôt être très largement adoptée. Le gouvernement français travaille sur une loi d’Affichage Environnemental obligatoire, qui consistera à apposer une note sur les produits et les services vendus.

Ce score sera défini par l’état, et sera basé principalement sur des ACV.

Proposition d’illustration pour l’affichage environnemental

On a là les outils pour aider les consommateurs à mieux choisir, et donc inciter les professionnels à mieux concevoir leurs produits et services.

Sources

Données d’impact environnemental de différents biens alimentaires (schémas en histogramme): https://ourworldindata.org/environmental-impacts-of-food

L’outil d’ACV simplifiée que je développe: https://karbon.earth

L’affichage environnemental réglementaire: https://www.ecologie.gouv.fr/laffichage-environnemental-des-produits-et-des-services-hors-alimentaire

La base de données utilisée pour les produits agro-alimentaires: https://doc.agribalyse.fr/documentation/conditions-dusage-des-donnees#guide-utilisateur

Exploit Airtable inside your React app

Wed, 11 May 2022 18:35:00 GMT

Intro

At Karbon 🌱, we iterate fast and we are always looking for ways to improve our products and services.

We had the idea of building an idea box, allowing users to create, share and vote for ideas of new features.

This is the typical project where we have to store unsensitive data entered by users. However we wanted to avoid impacting the rest of our application (e.g we don’t want to change our database schema).

Airtable is a great tool for this kind of usecase.

Let’s build this idea box ! 🗳️

Airtable provides an npm package to interact with their API. However, since I was working with multiple tables (one for votes, another one for ideas), I found myself repeating a lot of boilerplate code. I landed on a hook that seemed to provide a nice abstraction to interact with Airtable, but it had some bugs and the author didn’t seem to willing to spend time on it. So I decided to write my own hook.

On top of fixing a few issues, I worked on the following points :

provide an option to handle errors when fetching, updating or deleting data
provide an option to filter records using Airtable’s QueryParams

Usage 👩‍💻👨‍💻

Install the hook and Airtable’s package with your favorite package manager, for example npm :

npm install --save airtable use-airtable-crud

const { records, createRecord, updateRecord, deleteRecord, getRecords, loading } = useAirtable(
    'TABLE_NAME',
    AIRTABLE_API_KEY,
    'TABLE_BASE_ID',
    {
      filterByFormula: '{myField} = TRUE()',
    },
  );

You can find the source code of the hook here.

Outro

I hope this hook can be useful for you too! Prototyping and iterating is a great way to learn new things.

Atelier tech hunter

Wed, 07 Oct 2020 09:35:00 GMT

Expérience candidat

Résultat de toutes les relations entre l’entreprise et le candidat. CF UX, expérience client, etc

Les interviewers techniques sont un acteur clé de l’expérience du candidat vis à vis de l’entreprise.

Pour être les meilleurs dans l’expérience candidat : atelier pour être en constante amélioration.

Tenue vestimentaire : correcte. C’est pas très important.

Prendre connaissance du CV, préparer des questions par rapport au parcours.

Prendre rdv par mail/téléphone, proposer plusieurs créneaux, réserver une salle (si c’est en présentiel).

En visio: mettre la caméra. Utiliser Webex (pour être en conformité avec IBM).

Prendre connaissance du CV À L’AVANCE et le relire avant l’entretien.

En entretien on représente l’entreprise mais aussi notre équipe.

“Do’s and Don’ts” de l’entretien

Se présenter, s’assurer que tout va bien, papier, crayon, café, eau.

Présenter le poste, prendre le temps avec lui. Ne pas couper la parole.

Marquer positivement le candidat ! (parce qu’on se souvient toujours des très mauvais ou des très bons candidats)

CSS Interview questions

Sat, 15 Aug 2020 18:35:00 GMT

Those are just some random questions I sometimes ask when evaluating junior developers.

⭐

What will these lines do ?

.header {
color: red;
}

#header {
color: yellow;
}

.header > h1 {
color: blue;
}

input.header {
color: brick;
}

div.header {
color: brown;
}

<div class="header"> Hello, world! </div>

Name 2 CSS styling frameworks

⭐⭐

How would you build a responsive interface ?
What is the difference margin vs padding ?
What are CSS preprocessors ?

HTML Interview questions

Sat, 15 Aug 2020 18:35:00 GMT

Those are just some random questions I sometimes ask when evaluating junior developers.

⭐

Is there anything wrong here ?

<input type="text">
	<h1>
	Hello, world
	</h1>
</input>

⭐⭐

What is the basic structure of an html document ?

⭐⭐⭐

what is a web component ?

JavaScript Interview questions

Sat, 15 Aug 2020 18:35:00 GMT

Those are just some random questions I sometimes ask when evaluating junior developers.

⭐

What is the difference between == and === ?

What is the output ?

const animals = {};
['zebra', 'horse'].forEach(function(k) {
	animals[k] = undefined;
});
console.log(animals); ?

⭐⭐

What is the difference ?

[1,2,3].map(/*() => { ... }*/);
[1,2,3].forEach(/*() => { ... }*/);

What is the output ?

const numbersAndStuff = [0, 2, 4, 6, undefined];
const filtered = numbersAndStuff.filter(element => element);
console.log(filtered);

What is a closure ?
What is a Promise ?

What is the output ?

function updateUserName(user, name) {
user.name = name;
return user;
}
const john = {
name: 'John',
age: 48,
};
console.log('John', john);
const jane = updateUserName(john, 'Jane');
console.log('Jane', jane);
console.log('John', john);

⭐⭐⭐

What is the event loop ?

Nodejs Backend Development questions

Sat, 15 Aug 2020 18:35:00 GMT

Those are just some random questions I sometimes ask when evaluating junior developers.

⭐

What is Express ? What is a middleware ?
What are the Request, Response, and Next parameters in Express Middlewares ?

⭐⭐

What is an ORM or ODM ?
How can an API recognize a User ?

⭐⭐⭐

How do you handle authentication ?

React Interview questions

Sat, 15 Aug 2020 18:35:00 GMT

⭐

What is the difference between state and props?
How do you update a component’s state ?
What is the lifecycle of a component ?

⭐⭐

What are React hooks ?

How would you refactor this code ?

function Form() {
const [firstName, setFirstName] = useState('Taylor');
const [lastName, setLastName] = useState('Swift');

const [fullName, setFullName] = useState('');
useEffect(() => {
setFullName(firstName + ' ' + lastName);
}, [firstName, lastName]);
// ...
}

How do you handle a complex state ?

Let’s say I have a useState which in a parent component. This state represents complex data, and multiple children components want to update that state. What to do ?
What is “Context” ? What do you use it for ?
What is prop drilling ?
How do you handle code duplication in a React codebase ?

⭐⭐⭐

What are sources of memory leaks in React ?

How would you refactor this code ?

function ProductPage({ product, addToCart }) {
useEffect(() => {
if (product.isInCart) {
  showNotification(`Added ${product.name} to the shopping cart!`);
}
}, [product]);

function handleBuyClick() {
addToCart(product);
}

function handleCheckoutClick() {
addToCart(product);
navigateTo('/checkout');
}
// ...
}

https://blog.scrimba.com/react-interview-questions/ https://beta.reactjs.org/learn/you-might-not-need-an-effect

An elegant solution for memory leaks in React

Thu, 30 Jul 2020 18:35:00 GMT

🔴 UPDATE

This “solution” doesn’t seem to really avoid leaks. Even AbortController doesn’t seem to be the silver bullet against memory leaks 😰. Check out the discussion in the comments!

When working with asynchronous calls, for example API calls, you might have encountered this error :

Can't perform a React state update on an unmounted component. This is a no-op, but it indicates a memory leak in your application. To fix, cancel all subscriptions and asynchronous tasks in a useEffect cleanup function.

A GIF is worth a thousand words …

This is a small page that simulates some asynchronous logic on load, and then updates the view accordingly. Here, I unmount the component before the async work has been finished, and trigger the Error. (I took this example from this StackOverFlow post)

This is caused by this code :

function Example() {
  const [text, setText] = useState("waiting...");

  useEffect(() => {
    simulateSlowNetworkRequest().then(() => {
      setText("done!"); // ⚠️ what if the component is no longer mounted ?
      // => Warning: Can't perform a React state update on an unmounted component.
    });
  }, []);

  return <h2>{text}</h2>;
}

When running into that issue, I found multiple solutions, the most used one seems to be this one :

function OtherExample() {
	const [text, setText] = useState("waiting...");

  useEffect(() => {
    let isMounted = true; // 👈
    simulateSlowNetworkRequest().then(() => {
      if (!isMounted) { // 👈
        setText("done!"); // no more error
      }
    });
    return () => {
      isMounted = false; // 👈
    };
  }, []);

  return <h2>{text}</h2>;
}

But it requires you to add quite a lot of stuff into your component, having to deal with that isMounted variable all over the place …

There are other interesting solutions, like making your Promises cancellable.

You told me there would be an elegant solution !

I wasn’t lying! The solution I came up with is a very simple hook. It works just like React’s useState, but it basically checks if the component is mounted before updating the state !

Here is an example of the refactored code :

function OtherExample() {
  const [text, setText] = useStateIfMounted("waiting..."); // 👈

  React.useEffect(() => {
    simulateSlowNetworkRequest().then(() => {
      setText("done!"); // no more error
    });
  }, [setText]);

  return <h2>{text}</h2>;
}

Here is the CodeSandBox if you wanna play around !

TLDR

Use useStateIfMounted hook, that will only update the state if your component is mounted ! 🚀

I hope this might be helpful, feel free to reach me out in any case ! 🤗

Karbon Editor

Thu, 30 Jul 2020 18:35:00 GMT

Karbon Editor

What is it about ? 🌱

A web application that enables people from the food industry to understand the environmental impact caused by their activity. This is a collaborative tool that aims at making eco-designing food products easier.

My participation in this project:

Fullstack development (React, Node, GraphQL, Typescript)
Hiring, onboarding, training and coaching of team members
Project management
Management of UI/UX freelance workers
CI/CD

More info

Docker training

Fri, 13 Mar 2020 15:15:00 GMT

Docker Essentials: A developer introduction

Lab 1 : Running a container

What are Containers ?

A group of precesses, run in isolation. They run on a shared kernel. The isolation is provided by Linux namespaces.

VM vs Container VMs are heavy and slow to start because they have a full OS. containers can be directly run on top of virtual machines.

What is Docker ?

Docker is tooling to manage containers. “build once, run anywhere”

Why containers are appealing

no more “Works on my machine”
lightweight and fast
better resource utilization

Running a container

docker container run -t ubuntu top

The docker run command first starts a docker pull to download the Ubuntu image onto your host. After it is downloaded, it will start the container.

Even though we are using the Ubuntu image, it is important to note that the container does not have its own kernel. It uses the kernel of the host and the Ubuntu image is used only to provide the file system and tools available on an Ubuntu system.

docker container ls
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

        ubuntu              "top"               58 seconds ago      Up 58 seconds                           wizardly_germain

then

$ docker container exec -it 7d1a9d47e601 bash
root@7d1a9d47e601:/#

Using docker container exec with bash is a common way to inspect a Docker container.

Namespaces are a feature of the Linux kernel. However, Docker allows you to run containers on Windows and Mac. The secret is that embedded in the Docker product is a Linux subsystem.

2. Run multiple containers

running an nginx server :

docker container run --detach --publish 8080:80 --name nginx nginx

--detach flag will run this container in the background. the publish flag publishes port 80 in the container by using port 8080 on your host.

So here, go to localhost:8080 :

The name flag names the container.

Then, we can run another container :

docker container run --detach --publish 8081:27017 --name mongo mongo:3.4

We can see the containers by running docker container ls.

Removing containers

Stop the containers :
```
docker container stop <container-id>
```
Remove the stopped containers :
```
docker system prune
```

⚠️ Note : You should avoid using unverified content from the Docker Store when developing your own images because these images might contain security vulnerabilities or possibly even malicious software.

Lab 2 Add CI/CD value with Docker images

Docker images

Docker image is a tar file containing a container’s filesystem + metadata
for sharing and redistribution

Docker registry push and pull image from registry. Default registry: Docker hub. Public and free for public images.

You can also have a private registry, self-hosted, or cloud provider options.

Creating a docker image - with Docker build

Create a “Dockerfile”
- It contains a List of instructions for how to construct the container
docker build -f Dockerfile

Secret Sauce: Docker image layers

to make pushes faster, the lines in the docker file that change the most should be at the bottom of the file. Because when you invalidate a layer, you automatically invalidate the layers after it. So if it is the last layer, we optimize the caching system.

union file system
- merge image layers into single file system for each container
copy-on-write
- copies files that are edited up to top writable layer
Lab objectives :
- create custom image using a Dockerfile,
- build and run your image locally
- push your image to your account on DockerHub
- Update your image with a code change to see Docker image layering/caching in action

Create a python app

Paste this in a file called app.py to create a simple Flask app :

from flask import Flask

app = Flask(__name__)

@app.route("/")
def hello():
    return "hello world!"

if __name__ == "__main__":
    app.run(host="0.0.0.0")

then run

pip3 install flask
python3 app.py

Creating and building the docker image

Create a file named Dockerfile and add the following content:

FROM python:3.6.1-alpine
RUN pip install flask
CMD ["python","app.py"]
COPY app.py /app.py

The FROM line specifies the starting image to build your layers on top of it. In this case, you are selecting the python:3.6.1-alpine base layer because it already has the version of Python and pip that you need to run your application. he alpine version means that it uses the alpine distribution, which is significantly smaller than an alternative flavor of Linux. A smaller image means it will download (deploy) much faster.

It is best practice to use a specific tag when inheriting a parent image so that changes to the parent dependency are controlled. If no tag is specified, the latest tag takes effect, which acts as a dynamic pointer that points to the latest version of an image.

For security reasons, you must understand the layers that you build your docker image on top of.

The RUN command executes commands needed to set up your image for your application, such as installing packages, editing files, or changing file permissions. In this case, you are installing Flask. The RUN commands are executed at build time and are added to the layers of your image.

CMD is the command that is executed when you start a container. Here, you are using CMD to run your Python application. There can be only one CMD per Dockerfile. If you specify more than one CMD, then the last CMD will take effect.

This line copies the app.py file in the local directory (where you will run docker image build) into a new layer of the image.

It seems counter-intuitive to put this line after the CMD [“python”,“app.py”] line. Remember, the CMD line is executed only when the container is started, so you won’t get a file not found error here.

For full specifications about the Dockerfile :documentation here.

Then, we can build the Docker image.

docker image build -t python-hello-world ./

The -t parameter allows us to name the image.

then we can run

docker image ls

to see our image !

Run the docker image

docker run -p 5001:5000 -d python-hello-world

it returns > 0a8fa98d3b930c7b2d986c76fdeaf6b86cf041343be804a33585ccc9aedf5958

Navigate to http://localhost:5001 => you see “hello world!“.

To see the logs :

docker container logs 0a8fa98d3b930c7b2d986c76fdeaf6b86cf041343be804a33585ccc9aedf5958

It returns :

 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
172.17.0.1 - - [13/Mar/2020 09:10:34] "GET / HTTP/1.1" 200 -
172.17.0.1 - - [13/Mar/2020 09:10:34] "GET /favicon.ico HTTP/1.1" 404 -

The Dockerfile is used to create reproducible builds for your application. A common workflow is to have your CI/CD automation run docker image build as part of its build process. After images are built, they will be sent to a central registry where they can be accessed by all environments (such as a test environment) that need to run instances of that application. In the next section, you will push your custom image to the public Docker registry, which is the Docker Hub, where it can be consumed by other developers and operators.

Push to a central registry

docker login

Tag the image with you username. The Docker Hub naming convention is to tag your image with [dockerhub username]/[image name].

docker tag python-hello-world nansdibm/python-hello-world

Use docker push :

docker push nansdibm/python-hello-world

You can see your image on docker hub in the browser.

Deploy a change

Update app.py by changing “Hello World” with “Hello Beautiful World!“.

Rebuild the image

docker image build -t nansdibm/python-hello-world .

Notice the “Using cache” for Steps 1 - 3. These layers of the Docker image have already been built, and the docker image build command will use these layers from the cache instead of rebuilding them.

docker push nansdibm/python-hello-world

The push refers to repository [docker.io/nansdibm/python-hello-world]
bd3dadabc6d3: Pushed
455874a92037: Layer already exists
5f354b8b5dc0: Layer already exists
f61107386c17: Layer already exists
db49993833a0: Layer already exists
58c71ea40fb0: Layer already exists
2b0fb280b60d: Layer already exists

There is a caching mechanism in place for pushing layers too. Docker Hub already has all but one of the layers from an earlier push, so it only pushes the one layer that has changed.

Understanding image layers

FROM python:3.6.1-alpine
RUN pip install flask
CMD ["python","app.py"]
COPY app.py /app.py

Each of these lines is a layer. Each layer contains only the delta, or changes from the layers before it. To put these layers together into a single running container, Docker uses the union file system to overlay layers transparently into a single view.

The “copy-on-write” function is very fast and in almost all cases, does not have a noticeable effect on performance. You can inspect which files have been pulled up to the container level with the docker diff command. For more information, see the command-line reference on the docker diff command.

docker diff 0a8fa98d3b93

Because image layers are read-only, they can be shared by images and by running containers.

Because the containers use the same read-only layers, you can imagine that starting containers is very fast and has a very low footprint on the host.

Remove the containers

docker container ls

docker container stop 0a8fa98d3b93

docker system prune

Lab 3 Orchestrate applications with Docker Swarm

Container orchestration

What about production ? Automated scheduling and scaling, zero downtime deployments, high availability and fault tolerance…

What is container orchestration ?

Cluster management
Scheduling
Service discovery
Health management
Declare desired state
- Active reconciliation

Docker swarm, Kubernetes, … With hosted solutions :

IBM Cloud Container Services
Amazon ECS
Azure Containers
Google …

Setup a 3 node cluster with container orchestration with Docker Swarm.

Schedule and scale an application
Expose the application
Update with a rolling update
Demonstrate node failure and reconciliation

Create your first swarm

Using https://labs.play-with-docker.com/ Create a new instance

docker swarm init --advertise-addr eth0

To add a worker in this swarm :

docker swarm join --token SWMTKN-1-61p9nb8opn3tg8lchklkbyjz7uhysj8krjexrlaxpr3kaprte8-ex2t37ry9f8ux0xu92u3jfsm8 192.168.0.8:2377

To add a manager :

docker swarm join-token manager

You can think of Docker Swarm as a special mode that is activated by the command: docker swarm init. The —advertise-addr option specifies the address in which the other nodes will use to join the swarm.

With other instances, you can join the swarm by running

docker swarm join --token SWMTKN-1-61p9nb8opn3tg8lchklkbyjz7uhysj8krjexrlaxpr3kaprte8-ex2t37ry9f8ux0xu92u3jfsm8 192.168.0.8:2377

Back on node1, run

docker node ls

Deploy your first service

To run containers on a Docker Swarm, you need to create a service. A service is an abstraction that represents multiple containers of the same image deployed across a distributed cluster.

The —mount flag is useful to have NGINX print out the hostname of the node it’s running on. You will use this later in this lab when you start load balancing between multiple containers of NGINX that are distributed across different nodes in the cluster and you want to see which node in the swarm is serving the request.

docker service ls

docker service ps nginx1

Scale your service

Update your service with an updated number of replicas.

docker service update --replicas=5 --detach=true nginx1
nginx1

The state of the service is updated to 5 replicas, which is stored in the swarm’s internal storage.
Docker Swarm recognizes that the number of replicas that is scheduled now does not match the declared state of 5.
Docker Swarm schedules 4 more tasks (containers) in an attempt to meet the declared state for the service.

docker service ps nginx1

docker service logs nginx1

Apply rolling updates

docker service update --image nginx:1.13 --detach=true nginx1

Reconcile problems with containers

docker swarm leave

Determine how many nodes you need

For a production application, you should provision a cluster with multiple manager nodes to allow for manager node failiures. At least 3 manager nodes, but no more than 7.

3 manager nodes tolerate 1 node failure
5 manager nodes tolerate 2 node failures
7 manager nodes tolerate 3 node failures

Play with docker Templates :

Summary

Docker Swarm schedules services by using a declarative language. You declare the state, and the swarm attempts to maintain and reconcile to make sure the actual state equals the desired state.
Docker Swarm is composed of manager and worker nodes. Only managers can maintain the state of the swarm and accept commands to modify it. Workers have high scalability and are only used to run containers. By default, managers can also run containers.
The routing mesh built into Docker Swarm means that any port that is published at the service level will be exposed on every node in the swarm. Requests to a published service port will be automatically routed to a container of the service that is running in the swarm.

Lambda training

Wed, 11 Mar 2020 15:15:00 GMT

Learn AWS Lambda and see real world example with serverless

main documentation : https://serverless.com/

What is lambda

functions limited by time run on demand scaling is automated -> that’s why we call it “Serverless”

Pricing

pay per request
pay per compute time

AWS Lambda integrations

API Gateway -> to build a REST API
Kinesis -> processing data in streams
DynamoDB -> Managed NoSQL database
…

Example : Thumbnail creation

“Database Trigger-like” behavior for S3 New image in S3 -> trigger -> AWS Lambda function creates a thumbnail and push it to S3

/!\ the handler is the name of function that will be called in lambda The instructor only did a “hello world”.

The Serverless Framework

Aims to ease the pain of creating, deploying, managing and debugging lambda functions. Integrates with CI/CD tools. CloudFormation support (templates for deploying).

npm install -g serverless

IAM => Create a serverless-admin (new user). Programmatic access, and give administrator access. (not best practice though …)

serverless config credentials --provider aws --key XXX --secret YYY --profile serverless-admin

Hello world using the Serverless framework

serverless or sls => list of commands.

sls create --template aws-python --path hello-world-python

serverless.yml add

profile: serverless-admin
region: eu-west-3

sls deploy -v

the code is uploaded to s3, and then CloudFront is configured. IAM role is created for us.

Running a function from the command line

sls invoke -f hello -l

Updating the function

change the print statement.

sls deploy function -f hello

=> how to integrate that with a proper CI/CD system ?

Fetching function logs

sls logs -f hello -t

enables to stream function logs as they happen !

Remove the function

sls remove

add an IAM role statement to a function

iamRoleStatements:
  - Effect: "Allow"
    Action:
    - "lambda"
    Resource:
    - "*"

VPC for lambda functions

can launch a lambda in a VPC so that they can access EC2 instances or RDS instances. can specify securityGroupIds and subnetsId in the serverless.yml file.

Real world example : Service to create S3 image thumbnails

S3 events
plugins to deploy python dependencies (docker needed)

Triggering a function from an s3 event :

plugin : serverless-python-requirements it installs what’s written in requirements.txt file. Pillow => import PIL

Real world example : REST API with API Gateway and DynamoDB

Lambda functions are stateless, they don’t keep anything in memory, so they will write to DynamoDB.

package.json => like requirements.txt

environment :
  DYNAMODB_TABLE: ${self:service}-${opt:stage, self:provider.stage}

POST on /todos => call create function this is wired to the API gateway

under resources, you can specify the cloud formation template

response should always have a statusCode and a body.

Personal investigation : creating a full stack app

If you want to build an app that can leverage that rest API that is written in Lambda, you can use aws-amplify : https://aws-amplify.github.io/docs/js/api.

Formation AWS 09-10/03/2020

Tue, 10 Mar 2020 15:15:00 GMT

AWS Essentials

https://learning.oreilly.com/videos/amazon-web-services/9780134702186/9780134702186-awsa_01_01

1.1 Cloud Computing

On-demand Resource pooling (shared servers, shared buildings) Rapid Elasticity => growing when we need to grow Measured service => Pay as you go

1.2 Regions

Primary building block. VM, Storage, … You pick a region. Cost in AWS is determined by the region. You can choose multiple regions (for multiple audiences), or also to spread risk

1.3 Availability zones

Availability zones are a collection of datacenters. Connected with private fiber. us-west-2a us-west-2b us-west-2c … They are key to fault tolerance. System that’s resilient on the machine, but also resilient on the datacenter. What happens when a datacenter becomes unreachable.

1.4 Edge locations

Edge location Caching Amazon route 53 => DNS query as quickly as possible. Edge location serve Amazon CloudFront and they serve Amazon route 53

1.5 Scope of services

Global

AWS IAM AWS CloudFront AWS Route53

Regional

Dynamo DB …

Availability

Elastic block store Elastic compute cloud

1.6 Service overview

AWS console. top right => choose region.

Compute, Storage, Database, Networking, Analytics, Security, Management Tools, Developer tools, Internet of Things, Game development, Mobile services, … Rich environment.

2. Security in AWS

2.1 AWS Identity and Access Management

AWS IAM Authent Autho Users Group, Password Policy, Multi factor authentication => Smth you know combined with smth you have (phone (have), user/password (know)) Authenticating and Authorizing against AWS API.

Creating a user and group

Create a User

autogenerate a password AND require to create a new password at next sign in give the password to the person over an encrypted connection of some kind.

Create a group

Create a group, create permissions, and add user to the group

2.3 access keys

jane.doe password. She wants sdk, CLI, … Create Access Key for Jane.doe. Access ID, Secret Accesss key.

Permissions and Policies

Permissions are granted via policies Policies are written in JSON

We can genetare policies thanks to a tool created by Amazon.

All policies start with a default implicit deny.

We need an explicit allow to have permissions. Amazon Resource Name : Fromat pattern to find in the docs. arn:aws:iam:…:user/jdoe arn:aws:s3: arn:aws: smth can be let blank :::::

implicit deny => explicit deny => explicit allow

Statements specify : Principal (resource based policies) Actions

ex2:RunInstanced Resources
EC2 instances Conditions
time of day
from specific IP address
resource contains particular tag

2.5 Creating and attaching policies

Generator creates the appropriate syntax. Allow EC2 RunIntances, … (actions that are authorized). ARN : * (all)

And then you can attach a policy to a group. Developers => run EC2 instances.

2.6 Understanding Roles

Rotate credentials regularly. Credentials should never be shared, in the code or in environment variables Roles enable us to use temporary credentials that are cached by the cli or the sdk. high level of security

2.7 Creating roles

IAM => Role section create new role Role Name => Name of application. myApplication Role type (S3 stuff) attach policy

every application will be able to read from S3

2.8 Federated Users

We can create Single Sign on LDAP, Active directory.

We can federate applications so that they can all use the same S3 bucket. This way we can bypass backend APIs. Allow mobile apps to access directly s3 services.

User authenticate against our app Identity broker will make a acall to aws security token service. The service will return temporary credentials.

C’est pour gérer les devs je crois (?)

2.9 managing an MFA device

Multi factor authentication Hardware MFA device (to be ordered from Amazon :o ) Virtual MFA device QR code or show secret key. Scan. Add authentication code wait add second authentication code

have jane come over to my desk.

in case you can’t call jane to you desk

2.10 Resource Policies

Apply permission to aws permissions. policies that are not applied to users or group, but to resources. Make an s3 bucket publicly readable for example.

2.11 Applying resource policy

S3 service add bucket policy click policy generator allow getObject, listObjects … arn:aws:s3:::nameofthebucket/*

add statement

2.12 Using roles for cross account access

Payer Account (initial account) (pays the bill)

Development account - Production Account

you can leverage roles. You can allow a user in one account to access ressources from another account.

2.13 Best Practices

Leverage groups : give only ENOUGH rights. Not more.

Grant least privilege
Strong pwd policy
deny statements for added security
never share credentials
multiple accounts for isolation

master account

protect it
don’t use for day-to-day
delete default access keys
Enable MFA with physical key, and lock in a safe.

Networking AWS

Amazon virtual private cloud : VPC Segment networks Create a VPC in a particular region. Choose a particular address range We can create subnets.

3.2 Creating a VPC

from the aws console VPCs are specific to a region We have a default VPC from Amazon. 10.2.0.0 => the widest range of ip addresses

a vpc has a route table ACL => Allow incoming or outcoming traffic

3.3 adding subnets

Create a subnet below a VPC, and specify an availability zone We can assign a CIDR block (I really have to learn more about CIDR ranges !!) /24 => allows 256 IP addresses. The 1st 4, and the last one IP address are not usable (used by AWS)

3.4 Routing

ELB => load balancer EC2 application server Database Route tables that enable to hide EC2 and database from the internet. On premises network (what’s that ?)

3.5 demo

Routing subnets to the internet. vpc dashboard => route table

Route tables associated create internet gateway attach internet gateway to vpc

auto-assign public IP => do that to enable EC2 instances

3.6 public private and hybrid subnets

3.7 Network Access Control Lists

tool to use within VPC that are stateless like a firewall arround a subnet

3.8 Security Groups

firewall that applies to an EC2 instance. it is Stateful.

3.9 creating an NACL

web traffic acl associated to vpc

inbound => rule #100 HTTP 80 TCP 80 0.0.0.0/0 ALLOW rule #101 HTTPs 443 TCP 80 0.0.0.0/0 ALLOW

outbound => rule #100 TCP 1080-65535 ALLOW (send to the internet)

3.10 Creating a security group

web-server web application security inbound rules accept http and https from anywhere NACL => subnet can receive traffic Security group => Instance can receive traffic so this is not double work

3.11 VPC peering

It is a way to pair 2 VPCs. IP ranges must not overlap Peering connection PCX routing traffic from one range to another range

3.15 Aws direct connect

dedicated private connection 1Gbps or 10 Gbps options.

4 Computing in AWS

4.1 EC2

Elastic compute cloud (EC2) Virtual machine based on Xen hypervisor Various combinations of cpu, memory, disk, IO one VM called an instance

4.2 AMazon Machine Image

bit for bit copy of root volume you launch a machine from a machine image

4.3 Launching a linux instance

Amazon images Amazon Linux 64 bits .ebs => Elastic block

t2 micro auto assign public IP tag instance to keep your environment organized Name : Linux-demo Environment : dev Application : xxx

security group !!!

download a private keypair to to access the instance public IP and private IP are associated to that instance add security group to give shell access go back to the instance, change security groups, add ssh security group port 22 in ssh should be opened

ssh -i azez.pem ex2-user@public.ip.of.instance

4.4 Key pairs

public and private keys. Way of login into the operating system. 2048 ssh2 rsa keypairs

4.5 Instance metadata service

retrieve information about an EC2 instance service within an instance to get information about itself use this instance with scripts to bootstrap stuff. The script gets information about the instance to configure software.

169,254,169,254/latest/meta-data

4.6 Demo instance Metadata Service

curl http://169.254.169.254/latest/meta-data get the instance id : export instanceId= curl http://169.254.169.254/latest/meta-data/instance-id shell scripts that could access that variable.

4.7 bootstraping with userdata

it’s the key to maintain a self healing environment

4.8 Launching a windows instance

Meh windows

4.9 Stopping and terminating instances

Stop => can restart later terminate => throw away

4.10 Billing options

on demand vs reserved instances availability is not guaranteed on reserved instances

reserved instance have a commitment (1 year, 1 or 3 year)

4.11 highly available web application

3 tier architecture in amazon ec2

4.12 => 4.14 Introduction to AWS Lambda

Mobile Application could access Amazon Simple Storage Service. Amazon cognito => Federated user retrieves temporary credentials

upload directly to S3. Fire event to lambda function.

highly available and fault tolerant Speed time to market

5. Storage options

5.1 overview of aws storage options

EC2 => Instance store block storage that are built in to the EC2 instance storage there is EPHEMERAL. cannot take snapshots

EBS => not resilient to the loss of a data center. Independant Snapshots pay per gb per month

Amazon S3 write once, read many storage for the internet for things that need to be retrieved from users

AWS Storage gateway virtual machine that is designed to be run on-premises it exposes a device

5.2 Amazon simple storage service

object storage. highly available and fault tolerant. no filesystem Bucket => Object upload limit of 5 gb objects can be multi part up to 5 TB

object storage vs block storage

5.3 Demo creating buckets and object

host static website on s3 :o :o :o

5.4 Bucket Security with resource policies

5.5 Introduction to Amazon Glacier

Write once, read rarely glacier is cold storage 3-5 hours to get the files

5.7 Demo: Adding life cycle Rules

5.8 Instance Store Volumes

they are build-in and ephemeral

5.9 Elastic block stock EBS

it is not a NAS pay for provisioned storage data is independent from instance, and is connected over network can detach and attach to another

5.11 Creating an EBS volume in the aws dashboard

volume type (HDD, SSD)

6. Databases

6.1 Options

Amazon RDS => for sql stuff, mariadb Amazon ElastiCache => Memcached, redis DynamoDB => NoSQL, Event-driven. Redshift => Encryption

6.2 Amazon RDS

Snapshots, backups and patches Read replicas for when we have read heavy traffic

6.3 Data Durability

Multi AZ deployments Primary instance with secondary standby in a different availability zone

6.4 Launch an Amazon RDS instance

subnet group => several subnets publicly accessible => NO

6.5 Amazon DynamoDB

NoSQL Data store easy noSQL service Partition vs primary key

6.7 Scan and query operations

interface pour faire des query https://learning.oreilly.com/videos/amazon-web-services/9780134702186/9780134702186-awsa_06_07

6.8 Amazon ElastiCache

Redis => in memory database, backed to disk Memcached => in memory key-value store, not backed to disk

6.9 Amazon redshift

SQL compliant. It is a clustered service parallel queries across the nodes. Ideal for OLAP and BI apps pedabyte scale data wharehouse

6.10 database for e-commerce

Elastic load balancer => 3 EC 2 instances

Read replica(s) Multi-AZ => Amazon handles the primary and secondary thing

ElastiCache => Category pages, related products

copies to redshift in order to run BI stuff use redshift to understand conversion rates

7 Analytics in AWS

7.1 Real time stream processing => Amazon Kinesis

“SOAP” and old school technologies from back in the 90s. Use case : warn users of the lifecycle of a pizza before it is delivered to them.

Kinesis

7.2 Real time stream processing

It divides large amount of data into usable shards. (terabytes per hour)

1 shard => 1000 PUTs per second

Kinesis stream should be handled by Apps that have ONE SINGLE PURPOSE It can be Aggregation, Sliding Window analysis,

7.3 Big data with amazon elastic MapReduce

yet another “amazing” service.

Managed Hadoop framework (or spark, presto, hbase) provision single or thousands of instance.

for data intensive applications. Data mining, log analysis, scientific simulation, genomics. can read data from anywhere.

7.4 AWS Data Pipeline

Transform data from one data type to another datatype. Helps move data between data sources. It uses EC2 or EMR to transform Data. Can Execute SQL queries, or custom applications.

7.5 Video Subscription Service Business Intelligence

8 Developer and management tools

8.1 CloudWatch

Collect metrics on amazon stuff. It stores metrics for up to 2 weeks each service (EC2, ELB, EBS) will get its collection of metrics.

$/metric

Application-level metrics, or intance-level metrics.

CPU CloudWatch Alarms. Triggered on breach of threshold. Alarm CPUUtilization is over 80% for 2 periods of 1 minute for example

Doesn’t necessarily signal emergency.

can publish notifications. Up to 5000 alarms per account.

8.2 CloudWatch Logs

Search using specific syntax (see docs) Can search JSON fields Subscription filters

8.3 Cloudformation Infrastructure as code

Infrastructure as code cloud formation templates architectures are complex Manual process => bad documentation, difficult to reproduce Challenges with scripts => Dependencies

Solution : Automate with CloudFormation

Templates are meant to be written once and be deployed many times Parameters,

8.4 Cloudformation demo

designer within cloudformation to create templates.

Application Deployment and management

AWS Elastic BeanStalk Application management platform easy entry Ideal for developers. Super convenient Beanstalk handle resources automatically

AWS OpsWorks Configuration management platform supports chef recipes (configuration as code) based on chef recipes OpsWorks ideal for DevOps Engineers.

Launching an application on aws beanstalk

example

9. Mobile Application and services

9.1 Amazon Simple Queue service

Highly Available and fault tolerant. Seems like RabbitMQ Buffering events to enable loose coupling and asynchronous management

9.2 Amazon Simple Notification Service SNS

publish/subscribe

9.3 Amazon simple email service SES

Sending bulk email at scale Transactional emails, marketing emails, social networking get feedback about those who have been bounced, those who are marked as spam …

9.4 Amazon Cognito

Identity Integrates with major authentication providers. Enable SSO or login with social media Cognito can stor app data, state, preferences. Can store in local with SQLite.

9.5 Amazon Mobile Analytics

Makes it easy to measure usage of application. What are the going on trends? integrate AWS mobile SDK Also offers REST API.

10. High availability and fault tolerence

10.1 Elastic Load Balancing

What happens when this component fails ? Spread instances across availability zones

10.2 Listeners and SSL Certificates

Use the amazon certificate manager to have HTTPS !! Free

Upload the certificate to IAM, and then choose it for HTTPs listener

10.3 Load Balancing

Maintaining balance between availability zones

10.4 Creating an ELB

EC2 => load balancers define vps and listeners add security groups. The load balancer should have his own security group. Configure Health Check add EC2 Instances (they must be in the vpc we have chosen )

10.5 auto scaling

self healing services

10.6 Demand based Scaling

Scale to meet the demand. AutoScaling groups along with Amazon CloudWatch alarms. Scale down when CPU is low.

10.7 Creating an auto scaling group

create a launch configuration yum install -y httpd to be added in create launch configuration
Create auto scaling group select the right vpc configure scaling policies

11 Course Wrap up

11.1 The ideal

Highly available it is up a great majority of the time AMI, EC2, S3, Auto Scaling
fault-tolerant Aplpication continues to operate through fault AMI (?), EC2, Elastic Load Balancing
Secure AWS CLI/Console
Durable Data survives loss of infrastructure EBS, Snapshots, S3, Glacier

11.2 Best Practices

Design for Failure
- Everything will fail eventually
- any component can fail at any time
- “What happens when ”____” ?
Scale Horizontally
- Stateless Applications/Components (state should not be stored locally)
- ElastiCache, DynamoDB
- Distributed processsing :
- parallelize and batch
- Kinesis, Elastic Map Reduce
Disposable resources over fixed servers
Automate, automate, automate !
- CloudFormation
- Elastic Beanstalk
- AutoScaling
- CloudWatch
- Third Party Tools
- Ansible, chef, puppet as ways to automate environment
Security in Layers
- VPS, Routes, NACLs, Security groups
- IAM: Users, groups (least rights possible), roles, keys
- Leverage multiple accounts
- Protect Master credentials
Loose Coupling
- Microservices
- Failures should not cascade
- ELB, Amazon SQS, Kinesis …
Optimize for cost

11.3 The exam

80 minutes 55 questions Architect a solution that is technically appropriate, and cost effective.

Beyond Asset

Tue, 30 Jul 2019 18:35:00 GMT

Beyond Asset

(source)

What is it about ? 👷‍♀️

Beyond Asset is an asset management solution that manages and optimises the condition and maintenance of any infrastructure.

My participation in this project:

Technical dependencies management
Fullstack development (Angular, Node, MongoDB)
CI/CD
General troubleshooting

More info

Wattics

Sun, 30 Jul 2017 18:35:00 GMT

Wattics Dashboard

What is it about ? ⚡

Enterprise Energy Management Software for Smart Meters, Sensors. A user friendly dashboard to monitor energy consumption, and extract meaningful information out of tons of data.

My participation in this project:

Fullstack development (Ruby on Rails, JavaScript, React) with features such as white labeling, costs calculation, localization, …
UI redesign
Automatic testing

More info