Author Archives: Carles Mateo

News for the blog, upwards and onwards

2020-03-06 Heya, I’m doing a set of improvements to the blog.

One, you can already see. I added a new section to the CSS @media, so now screens bigger than 1,800 px in width, will use that width for rendering the page. The original WordPress theme at 960x was too small for our current screens. I will add a new CSS @media for 4K screens promptly.

Other is about the organization of the content. I want to separate a bit the contents, now articles are sequential and is difficult to discover nice contents if they have 2 or more articles more recent, so I will group articles by content and provide a small index on the top page. Also I will provide more areas for Operations, SRE, where it will be easy to locate code, scripts, tricks… things that are useful to our day to day. I also want to make visible the articles about living in different cities, for IT Engineers, with useful tricks and tips. And keep the more complex and more interesting Engineering matters in the main page.

Updates

2020-03-13 15:49 Added SSL to the blog

With more delay I wanted, I bought a SSL certificate, configured Apache, and after few changes to the blog has been set. One very annoying is that WordPress linked the images statically pointing to http://blog.carlesmateo.com so I changed the latest article’s images to point to relative path so they will work nice with http or https.

My reflection is that everything negative can have its positive output. With this coronavirus thing, I decided to focus into improving things. And so I’m doing. :)

Working in Barcelona, Catalonia, for IT Engineers

Created in: 2020-02-20 by Carles Mateo Last Updated: 2020-03-13 19:29 UTC

Note: This document is under construction

Society, Politics and laws

Barcelona is the capital of Catalonia, a Nation with a strong desire of freedom, that wants to get the independence from Spain (which invaded militarily in 1714 and bombed the city many times). Catalans are very different in small villages and in the city center, but me as Catalan, I think that we are nice but reserved, because we are used to be prosecuted just for being Catalans, so we people don’t earn our trust easily. However, if you make a Catalan friend, our loyalty is like family. This is related since medieval books already.

When we suffered the Spanish invasion in 1714, and later the invasion of dictator Franco with the help of Hitler and Mussolini, the Catalan language was forbidden, prosecuted, universities closed, Catalan Constitution removed, Parliament closed… but part of the freedom was recovered in 1975 when the fascist dictator Franco died. Most of the Catalan particularities were forced into assimilation to the Spanish model, however one interesting thing that was not changed was the Civil Right. That’s very important cause some laws survived and are different from the Spanish laws, and from most of the places in the world. One worth to mention is that our Civil Law gives us “Separation of Goods” (separacio de bens), and that means, that if two people marry and one has 3 houses and €20,000 in the bank, and the other has €500 in the bank and nothing else, if they divorce, they’ll get exactly what they had. So individual A will keep their 3 houses and €20,000 and B €500. This is very different from the “Earning Goods” (bienes gananciales) in Spain, where if they divorce A and B would get exactly half of everything. When they get married is like everything goes to the common pool.

Must say that there are details, like if the partner leaves the job in order of take care of the house, they may get something in compensation, or if there are children, at least one house will be for the mother if taking care of the children, and the father will have to pay a pension to feed and take care of the children. Still, it seems to me much more fair that what I saw in most of the countries (even if in some they have pre-marital contracts).

The Catalan society is hard worker and loves to read and to study. Compared often to the Jews, as an insult, by the fascist authorities in Spain during the dictator.

One show of the passion for reading can be manifested by Sant Jordi’s celebration, the 23th of April, which is a day in which girls gift a book to the men, and men gift a rose to the women they love, or they family or female friends.

The Castellers or Human Towers are declared World Heritage and have been present in many international celebrations like Olympic Games.

And the traditional parties in the villages have many books talking about them and are epic.

As Catholics, it is typical to have “el pesebre”, a recreation of the birth of Jesus, but our irreducible character have has to have particular elements like “el Caganer”, which have became world wide famous few years ago.

Pesebre de Nadal

The love for Freedom and the defense of Human Rights has been a characteristic tread of the Catalan culture since it’s origins, around the year 8th Century and the first bilingual dictionary known is Catalan-German German-Catalan. Catalans also invented the navigation code for ships that became universally accepted. Probably this loves for Freedom is what makes Barcelona and Catalonia a place full of entrepreneurs and small business owners. It is said that “Catalans, from the rocks they make bread” (els Catalans de les pedres en fan pans) about the inventive we have to detect opportunities where others see nothing.

The 11th September is the national day, and is the day the Catalans remember that they fight to the end for their freedom that 1714, alone, against Spain and France troops. That day every year, there are many cultural acts, and there is a demonstration for Independence that some years got to 2 Million people, from a total population of 7,5 Million. You can visit parts of the city where there are still holes in the walls from the different times Barcelona has been bombed, at different ages, from Romanic era to 20th century.

Weather

The weather in Barcelona use to be excellent. Sun is present from 7 in the morning to 20:00 almost all the year, and specially in June, July and August, people enjoy the beach.

It has a weather similar to California.

It rarely snows, but if you go to the North, near Pirineus montains, you can see snow in winter.

Even if weather use to be excellent, with no wind at all, and few frequent rain, must be said that when it rains it rains. Rain can last 4 hours and more, and be very intense. Very often the semaphores (traffic lights) stop working in Barcelona when it rains.

The city

The city is very well designed with most of it being square islands similar in size. Is easy to drive as every street is two ways and you can go back by just turning right right right right or left left left left.

With a beautiful architecture thanks to genius like Gaudi, or Dali (both put in jail for talking Catalan at different times) .

Sants street is the longest commerce way, in Europe they say, created by many small commercial stores. Many of them have been bought or replaced by Chinese, as the traditional owners could not afford the more and more higher rents for commercial locals in Barcelona. Still many resist.

Transport

The city has buses, metro, taxis, tram, bicycles that you can use paying a yearly sum.

Huber was forbidden by the Spanish government.

Taxis in Barcelona are very characteristics, all the same color, black and yellow. In the airport there is an official taxi stop, so don’t get to any random guy that asks you if you want a taxi. The taxi drivers are in a specific place and they will not go to you, as they expect you to go to them.

In the city you can stop a taxi just by raising your hand horizontally, but only the cars with a green light turned on are available, the other have a customer inside and will not stop.

Airports

Local food

One typical food. “La calçotada”. Which is a social party, normally for lunch where we eat scallion/spring onions and meat cooked at the fire on the barbecue. We peel the external layers, and eat with a sauce called “romesco”.

Calçots, scallions grilled at the barbequeu
Butifarra d’ou (egg sausage, to eat like that)

Population

Languages

Religion

airbnb and hotels price

The controversial mayoress has forbidden airbnb in Barcelona. It is only allowed to red a room, but not the entire flat.

Hotels prices range from €110 to €175 per night, except when the city is full and prices double or triple. For example for the Mobile World Congress.

Price of the rent

In the recent years rent has increased a lot.

In the year 2020 a flat with furniture, 5 years old new, 80 square meter in a middle class zone, can cost around €1,200/month.

English and expats

Well, it’s a shame. In Barcelona few people talk a decent English or even understands it.

You better print the address of the places you go to show to the taxi driver.

Salaries for IT

Barcelona is one of the most attractive places in Europe for IT.

A low cost of live compared to major cities like Berlin, London, Edinburgh, Cork or Dublin, and salaries very high for IT, with the excellent weather, a market of 40 millions Spaniards and 7.5 million Catalans (around 10 if we count Catalan speaking zones in the Meditarrean sea).

The minimum inter-professional salary is (updated to Jul 2019) €1,050 net per month.

Salaries for a Senior Software Engineer with 5 years experience will range between €35,000 and €55,000 gross per year. Most of companies provide an optional bonus too.

€55,000 gross per year is approximately €3,267 net per month within 12 payments in a year.

Internet

Internet is decent, most areas of the city have Fiber and in some places they have 1 Gbit/s connections.

In some places far from the core of the city only DSL arrives.

Office Material

The paper we use here is Din A4, like in Ireland.

Buying Hardware

Barcelona is full of Hardware stores. There is one street where they are concentrated, is called Sepulveda street.

Security

Police

Nightlife

Dangers/Terror histories/Bad experiences

There are many to tell, I’ve pending explaining seeing thieves in the airport, fake taxi drivers offering their services, pickpocketers in the center, skinheads robbing in the metro… don’t ever leave your belongings unattended.

LGTB+

Restaurants

Those are my recommendations.

La parra

I come here since I was a child

http://restaurantlaparra.cat/

Santamasa

http://www.santamasarestaurant.com/

Centric 35

Is from a good friend of mine 😊 The Cheff and owner. If you tell Oriol that you’re a friend of Carles he will be very happy.

https://www.tripadvisor.ie/Restaurant_Review-g187497-d3635494-Reviews-Centric_35-Barcelona_Catalonia.html

https://restaurantcentric35.business.site/

Clarius (Catalan-French cuisine)

https://hedofoodia.blogspot.com/2019/05/clarius-restaurant-barcelone.html

Can Xurrades

http://www.canxurrades.com/


1881 per Sagardi

Restaurant and terrace in the Museu d’Historia de Catalunya
Plaça de Pau Vila, 3, 08003 Barcelona
At the top of the Museum of History of Catalonia you can enjoy a restaurant, a coffee bar, and a terrace for having cocktails. The terrace is a cityview where you can see all the port and part of Barcelona. Is very beautiful at nights. There are few tapas bars and pubs and open discos just down next to the Museum’s entrance.

http://www.mhcat.cat/la_visita/mirador_restaurant/restaurant_1881

The museum is very interesting too.

1902 Cafe Modernista

Next to the Recinte Modernista Museum, you can have tasty tapas with beautiful views and after going to the museum.

Carrer de Sant Antoni Maria Claret, 167 | Recinte Modernista de Sant Pau, 08025 Barcelona
(st Sant Antoni Maria Claret, 167)


https://www.tripadvisor.ie/LocationPhotoDirectLink-g187497-d6503265-i137778634-1902_Cafe_Modernista-Barcelona_Catalonia.html

The terrace is owned by the restaurant which is inside, and is very nice, and the menu looks amazing.

Iconic places in Barcelona city

Sagrada Familia from Gaudi.

F.C.B. Football Club Barcelona stadium “Nou Camp”.

Nearby Iconic places

Monserrat, has unique type of rounded mountains. And on top, in the church, there is “la Moreneta” which is a figure of the Mother of Christ, which is black.

Activites

Working in Cork, Ireland, for IT Engineers

Created in: 2020-02-20 20:20 by Carles Mateo Last Updated: 2020-03-08 19:15 GMT

Note: This document is under construction

Society, Politics and laws

People are very polite. One nice thing is that you say Bye, thanks to the bus driver when you go down.

The Irish society is very welcoming and also participate a lot in charity. You can see how many people are running in races for a charity cause, and how many commerces collect money for charity causes, from pubs, to supermarkets.

I like the Cork Zombie Walk, where people dress as zombies, ghosts, etc… and walk over the city and finally they go to drink beers and have dinner together. You can volunteer to makeup people and is a very funny experience. Money collected goes to guide dogs for blind people.

They are a very polite society and even in the queue of the supermarket will ask you to any single customer: “how are you?”. At the beginning I was explaining all my life.

Weather

The weather is one of the most common form of claims people do. We use to say that we have 4 seasons in a day. It can rain for 15 minutes, be sunny, be windy, snow, be sunny again, rain… 10 times per day!. Although is not very usual that it snows, and heavy wind is neither usual. Rain is really a tiny courtain of water, so I don’t even bother. Having a jumper with a hood is enough.

You can get a top-up prepaid simm for 10 €. It is what I recommend to my American friends and colleagues when they visit Ireland.

The power plugs are like in Scotland and England. So you’ll need a transformer. There are universal ones that costs between 4€ and 10€.

I was surprised to discover that the English talked in the streets, and the practical English, is much more fluid and similar to Catalan grammatically than the official English that the schools teach.

For example, instead of asking: Did you go to the cinema? Is perfectly valid to ask: went to cinema?.

The Irish are very polite, and they respect the physical distance. If they are stood in front of you in the supermarket they will say sorry, and everybody says “Bye, thanks” when going down the bus. From Cork to Dublin you have less than 3 hours driving, I use more as I have a lesion in my back and I stop often, and 3 hours by train. There are also buses connecting Cork and Dublin airport every half an hour almost 24×7. Companies are: aircoach.ie and gobus.ie

The Irisih will talk you randomly for no reason at the pub, or at the street. They have good sense of humor, are near, caring, nice and talkative. And they help others.

Once my Visa card failed on the cashier in the supermarket and the man behind me, thinking that I was in trouble, offered to pay my food. That really touched me.

Also another time I entered into the pub with paper bags from the supermarket, and due to the rain, one just crashed and my products when to the floor. One woman in his 65s jump from the chair and came running with a plastic bag “if it’s of any help”, offered it to me. I almost cried. I still keep that small blue bag.

The city

Cork has some of the better universities in Ireland, and in Europe.

So many students from all Ireland come here, and also international students.

The most well known universities for IT are UCC, more focuses on the theory, and CIT, more practical. So seasonally is a stuadent’s city.

Transport

Cork has a service of buses that when they work, they work well. Unfortunately often they have to be fixed and you may miss one or two because they did not pass or bass indicating that they’re going for maintenance.

Cork has a train that has good reputation (I’ve not used it), and will bring you to Blackpool, or to Cobh, or to Dublin.

There are long distances buses aircoach.ie and gobus.ie that travel to Dublin almost every half an hour during the day or the night.

If you drive at 9AM traffic is terrible, as parents go to drive the kids to the school.

A very convenient app is Free Now, formerly MyTaxi. It will show you in real time where the taxi is, who is, you’ll be able to pay with Visa… Taxis in Cork or Barcelona are not like in the States. They don’t incorporate a monitor so you can see your route, and many taxis also don’t allow to pay with credit card (that happened to me in London too).

Take in count that from 08:30 to 09:30 you will need three times the time required to travel to a destination, as parents bring children to the school and people go to work and the ways collapse.

Also take in count that if you call a taxi using Free Now at those high intervals, maybe nobody will answer your request, or maybe they do and you’ll have to wait 30 minutes. Be advised.

If you are American you can drive legally with your American license for a year, after you’ll have to pass the exam, insurance will be €3,000/year.

Driving: We drive by the right side (as opposite to the wrong side) :)

Driving in the right is like in Scotland. Myself didn’t find it specially difficult but is much more easier renting an automatic car. I do this, and so I don’t have to think about the Gearbox when I travel by the States, or visit Barcelona or other European countries.

There are few automatic cars available. Absolutely the majority of cars in Ireland use manual gear. Don’t ask me why.

If you want to rent a car in the airport it will be around €40 per day, so ~€1,200 per month. I know that you can do good arrangements with small companies up to 650 € if you rent all the month. If you buy a car, insurance is really expensive. If you come from the United States you can drive a rent car, but if you want to buy one you will have to do the exams to get an Irish driving license. Insurance is particularly expensive, with prices that could be as insane as €4,000 if you don’t have a no claim bonus. If you are American they will not accept your American no claim bonus. Some of the insurances wants you to buy in one go, while others will allow you to split the payments in 30% in advance, and then every month. Knowing the tricks I was able to get my insurance for €1,250.

Airports

Cork airport is small, but comfortable. The bad point is that they don’t have many direct flights. Dublin has. We have direct flights to Barcelona, but seasonal, from end of March to end of October, with Aer Lingus.

Normally you’ll go to Dublin airport by aircoach.ie or gobus.ie and fly direct to Barcelona, or you will fly from Cork to London to Barcelona, or from Cork to Amsterdam to Barcelona.

There is one ferry to Santander, Spain, and another to France.

Local food

Food in Ireland has very good quality. In Cork there are many farmers, and is not strange to see the photo of the local farmer and the address of the place, when you buy products in the supermarket.

Meat is very good, the milk, and the bread which last only few days.

The most famous is probably the Irish breakfast, very similar to the Scottish breakfast including sausages, red beans, pudding (in Scotland they call it haggish), eggs, rashers (bacon), a potato triangle

This is actual breakfast from the cantine at work

I just discovered today that is like a tradition doing on Tuesday pancake (what we call a crep in Barcelona).

Population

Cork is the second biggest city in Ireland, with a population of 280K. However being the 2nd doesn’t make it impressive in terms of number of citizens. Ireland is a small country, with only 4M inhabitants, and that you can cross from the bottom to the top, by car, in around 4 hours (I counted the Northern Ireland here), but this is a reason why people live very well.

Ireland is full of green and nature, and if you are outside the city you can notice how pure the air is in here.

If you compare to the F.C.B. “Camp Nou” stadium in Barcelona, with 99,354 sittings, or to the 2 Million people demonstrations for independence in Barcelona, you see that is no that much.

Languages

Official languages in Ireland are English and Irish (Gaelic).

All the official messages and driving signs are bilingual.

Cork citizens have the well deserved fame of having the most difficult English in the world. And it’s true. It was not only me at the beginning, there are Americans that don’t understand the accent too!. Basically it has a characteristic music and they remove all the spaces between the words, modulating also the volume of the sound. If that was not enough, they have their own slang and there are many books with that slang, so don’t make me guess how many words in the slang they have!. I can tell you that they have phrases, that, depending on how they pronunce have antagonist meanings.

Here we say runner, for running shoes, rashers, for bacon strips, we say grant or it’s grant to say that is Ok, and heya, thankya, bye now… not to bad, to express we are fine, it was mank (this food was horrible), this was deadly or class (like terrific), sound (thanks. Can also say he is sound), story or what’s the crack to ask how is going, he has notions (he believes is better than the others because he bought a car, or a big tv) and if you hear tikety boo (all Ok), or oki doky karaoke don’t be surprised. :)

We also use jump the gun (come to a conclusion too quickly), fair enough, and in my work we say informally go bananas or go coocoo although I think I’ve heard this to the Americans too.

I love specially the expression thanks a million, that actually comes from the Irish (Gaelic) language. And if you’re asking yourself, few people talk it fluently. Most of people have notions as it is taught in the schools, but not many families speak Irish at home.

Religion

Religion is Catholic, and they practice and only in 2019 was voted in referendum to approve the abortion.

airbnb and hotels price

Airbnb is legal, and people uses it. Myself I was host in my flat for a while and was a great experience.

I had a lot of people from Dublin, that came to Cork for a concert or a wedding, and they just need the room to sleep, one day or two, basically.

Hotels price is like Barcelona between €100 and €150 euros per night. This price can be much higher if there is a congress. There are many seasonal activities where the city is full.

Something very typical to use is the B&B, Bed and Breakfast. B&B is a big house, normally managed by a family living there, that rent rooms, like if it was a hotel. Many Irish use it when going to a wedding or to visit the family and are very popular here. Typical prices are €100 or €50 euros per night. That’s typical in Scotland as well.

Finally the hostels, the cheapest price I have heard is a bed for €17 per night, but in a room shared with 7 more people.

Price of the rent

Price of the rent has been increasing like crazy in the last two years. A 2 bedroom flat, with 2 bathrooms, and furniture will cost at February around €1,500/month.

People live in houses, more than flats. Except for the city center, of course.

If you don’t mind to live in a house, far from the bus and the city center, probably you can get a house with 3 bedrooms and 1 bathroom per 1,300/month.

You need references. That’s very important.

The best website to find properties to rent, or rooms, is https://daft.ie

The usual is to leave 1 deposit and pay the current month, if the Landlord or Landlady hired an agency, they are the ones that will pay the agency.

English and expats

Some of my colleagues are afraid to go to an English speaking country cause they feel ashamed, they think their accent is not very good.

Well, the English of Cork is difficult, but they know and will speak to you slowly if they see you have difficulties. Don’t be afraid to say “Say it again” “Sorry, can you repeat?” “Did you mean xxxx?”.

In my first job here most of my IT colleagues were expats: from France, from Italy, from India, from Romania, Kurdish, from Nigeria, from Spain… all of them have a different accent and we understand each other. And if you go to meetups (I recommend meetup.com in order to know people) or conferences, you’ll see that many have a terrible accent worst than you. Fortunately English speaking people understand that English is not native language and value your efforts. If you ask them to correct you, you’ll learn and improve very fast. We’re very lucky to be appreciated for our intelligence and not for how we dress, or for our accent.

And many times locals will fins your accent very exotic and interesting. :)

In my second job there is not a single Engineer in Operations Irish. I’m Catalan, there is Italian, Indian, German, Russian, Romanian, English, Danish…

So Ireland is very welcoming, and is hungry for IT experts.

Salaries for IT

90% of IT companies in the world have presence in Ireland. That’s massive.

Here, in Cork, you have nice companies like Blizzard, Ibm, Quest, Dell, Logitech, Qualcomm, Vmware, McAffee, Trend micro, Facebook…

For people not in IT, there are several companies that provide Call Support in different languages, like Apple, VoxPro… so many non IT people work there.

Salary varies a lot depending in which role you work and if is a big company.

For the first multinational I worked here, a grad, which is basically somebody that finished the degree and has two years of experience, or that finished the degree and did one master, the salary was €30,000 gross per year plus 20% optional yearly bonus.

A Senior Software Engineer with 5 years of experience developing Software will get between €60,000 and €75,000 gross per year. Salaries in Dublin are around 15%, or more, higher but the rent it is too.

As Cloud Architect I had offers of 6 figures.

The minimum wage for an adult at Feb 2020, is €10.10 per hour, that’s around €1,656.20 net per month.

Internet

Internet is good and many places have fiber.

Myself I live in a quiet area, 15 minutes by car from the center, and I have a 360 Mbit/s fiber connection.

In some places they have 1 Gbit/s connections.

As people tend to live in houses instead of flats (the city center is really small) many places can only use DSL or poorly 4G connections, which is very annoyng during covid-19 lockdown and WFH as we have to disable video in chats with the colleagues.

Office material

The paper we use here is Din A4, like in Catalonia.

Buying Hardware

There are two Hardware stores in the City Center. The rest you will have to order using Amazon.co.uk.

If you look for an UPS (Uninterruptible Power Supply), you may see that most of them are not shipped to Ireland. This is due to restrictions flying batteries.

There are specialized sites in UK where you can buy and they will ship them to Ireland.

Security

I consider Cork a very safe city. Starting a fight is a crime, and because their polite personality, violence of fights are not the usual. Exception to this can be found in the city center, late at night, with young (and not so young) people drunk.

People, men and women, walk fearless by the street at night.

Must say that the situation has gone a bit worst since the last two years, cause heroine has arrived to the city, and that causes some addicts to act in desperation. There are really few cases. Any tragedy or violent crime appears in the newspapers and is a new for a week, pointing, that fortunately this is not frequent.

People leave their belongings in the coffee and in the pub while they go to the restroom or to smoke. This is something you cannot do in Barcelona.

However I have known of some foreigners stealing jackets in pubs like Craig Lane. It is a shame that they do this in a so nice country like this.

It is super safe specially if we compare to Dublin, where I’ve seen yonkies running and taking the phone from people talking, in the street, or going in bicycle and stealing by pulling the cameras on the neck of Japanese tourist (that I’ve been told, I didn’t see). I don’t like to go to Dublin for that reason.

Police

There is one kind of police only, the Gardai.

They don’t carry guns or batons, as Irish consider that this scales the violence. That if police has guns, bad guys will have machine guns, etc… So basically they try to convince the offenders to stop their attitude and surrender. They have a unit of weaponized gardai for emergencies.

My interactions with Gardai have been very positive. They have been very nice, polite, attentive. Coming from Barcelona where Spanish police hits the voters of a referendum, or hist the Catalans citizens, just cause they are Catalans finding normal policeman at the service of the citizens is something I really value.

An American friend of mine that was police, adds that wearing a knife is illegal.

Nightlife

The pub close around 1:30 AM, some at 00:30. When they do the “last call” means that is the last 5 minutes you can order drinks. After that they will start cleaning and you can stay while they do, normally for half an hour or an hour more, finishing your drinks. So yes, when it’s last call some people order 3 beers.

Almost all the pubs in the city center play live music almost every night. Few specialize in traditional Irish dance, most are modern rock singers or bands.

There is a disco that closes late, is called “Vodoo rooms”, and has 3 different kind of musics / ambiances with a terrace on top where is it possible to smoke. It is very well dimensioned in terms of security, and personnel is all the time attentive to detect liquid on the floor or broken glasses, and the clean everything very quickly and effectively.

Security guards in the pub have a band with their photo, name, and license number. Only people that passed the certification can be security guard. Many of them are people in their 55, so it’s less likely that they’re going to cause trouble to customers. In fact I found them very attentive, telling me hello and bye every time.

Dangers/Terror histories/Bad experiences

As said I don’t have the perception of danger in Cork, however, there is a warning signal that things can be wrong with youngsters.

This is something than an American expolice and I discussed, and he had the same impression. I have discussed with other adults and see the point like me.

The young guys, between 15 and 17 y.o. go in gangs, in groups 10, 15 or 20, and at that point they can be rude, disrespectful, have fun at the expense of others or damage a property for fun. This is the biggest danger we see.

Coming from Barcelona I laugh at the attempts of some young guys in group to pretend to be though and intimidating, so no big deal, but I had a bad experience twice, when I was walking in the street and a group throw me an egg, from behind from a running car.

I was walking using my headphones when suddenly I saw the egg passing by my side and exploding on the floor, half meter in front of my feet, and the car running away. They missed it. But as I continued walking I saw more eggshells and egg contents on the floor, so they were targeting random people for fun. That was when I had been for a year in Cork, and I decided that this will not ruin my positive experience.

I talked to my boss and he told me that this is something relative popular that some youngsters do, and that he used to find it funny when he was a young boy (he never did) until he knew about the case of a woman, that just turn back at the same time that the egg was coming from behind, at high speed as coming from a car, and impacted in her eye, making her loss her cornea, so the eyesight of one eye.

The second time I was walking on the same street, 2 Km distance from the when the first time, I was in front of the hospital, walking to the gym. I was walking 3 hours every day, and spending 45 minutes more on the bicycle in the gym, every day. As I walked my way to the gym, passing in front of the hospital, with my headphones and my hood, as it was cold or slightly raining, I noticed a heavy impact in my right arm and I got scared and my reaction was to get defensive to fight back. In slow motion I saw the egg bouncing from my arm to the floor. It didn’t explode on contact with my jacked and arm, bounced, and exploded in the floor without getting to me. As my reaction was to get scared by the impact in my arm, and my primary reaction was to fight back, they were lucky that I was talking by phone with a relative, and I explained what happened, which prevent me from my impulse to run after the car. They were lucky, cause 200 meters ahead they had to stop cause there was a semaphore and cars stopped by. They could not have escaped if I chased them and hit the windows. Fortunately nothing of this happened.

There is a community of poor people that live in tents. That’s very sad. Here is really cold, under 0 centigrade degrees some times (today 29th Feb we are at 2 cent. degrees. There are young people also in there, I read about one case in the newspapers. They could have died out there. Social services are insufficient according to the locals and they organize charity to help them live in a better place, provide better clothes, etc… Those people living in tents are exposed to drug addicts that rob the place trying to find money or something to sell.

This is the dark side and many of my expat colleagues do not know this. I live in Cork since 3 years ago, I’m well integrated, I read the newspapers, have local friends, ask the taxi drivers… so I know.

LGTB+

I would say that Cork, even being Catholic, is open to LGTB+. For St. Patricks cellebration some drag queen are in the carousel.

There is at least one bar/disco declared LGTB+ Friendly. Is The Chambers.

They do very nice concerts on Saturdays, and on Fridays they do Karaoke.

Restaurants

You must be advised, that unless specified different, restaurant will close the kitchen at 9:00PM (21:00).

Cofe places close around 6PM, at 8PM in the city center with few exceptions. (In Barcelona I used to go to a coffee that closes at midnight or later)

Most of the pubs serve food until 21:00 too.

Iconic places in Cork city

The English Market


Iconic nearby places

There are many precious walking paths and parks, full of green. The weather may surprise you, so it’s important to wear good clothes with a hood.

Gougane Barra http://www.gouganebarra.com/

Cobh

Kilarney

Kilkeny

Activities

www.runireland.com/event-listings/

meetup.com

Cork Zombie Walk

Cork MidSummer Festival

CTOP.py

Current version is v.0.7.6 updated on 2020-07-29 20:27 IST (Irish Standard Time).

Find the source code in: https://gitlab.com/carles.mateo/ctop

Clone it with:

git clone https://gitlab.com/carles.mateo/ctop.git

ctop.py is an Open Source tool for Linux System Administration that I’ve written in Python3. It uses only the System (/proc), and not third party libraries, in order to get all the information required.
I use only this modules, so it’s ideal to run in all the farm of Servers and Dockers:

  • os
  • sys
  • time
  • shutil (for getting the Terminal width and height)

The purpose of this tool is to help to troubleshot and to identify problems with a single view to a single tool that has all the typical indicators.

It provides in a single view information that is typically provided by many programs:

  • top, htop for the CPU usage, process list, memory usage
  • meminfo
  • cpuinfo
  • hostname
  • uptime
  • df to see the free space in / and the free inodes
  • iftop to see real-time bandwidth usage
  • ip addr list to see the main Ip for the interfaces
  • netstat or lsof to see the list of listening TCP Ports
  • uname -a to see the Kernel version

Other cool things it does is:

  • Identifying if you’re inside an Amazon VM, Virtual Box, Docker or lxc
  • Uses colors, and marks in yellow the warnings and in red the errors, problems like few disk space reaming or high CPU usage according to the available cores and CPUs.
  • Redraws the screen and adjust to the size of the Terminal, bigger terminal displays more information
  • It doesn’t use external libraries, and does not escape to shell. It reads everything from /proc /sys or /etc files.
  • Identifies the Linux distribution
  • Shows the most repeated binaries, so you can identify DDoS attacks (like having 5,000 apache instances where you have normally 500 or many instances of Python)
  • Indicates if an interface has the cable connected or disconnected
  • Shows the Speed of the Network Connection (useful for Mellanox cards than can operate and 200Gbit/sec, 100, 50, 40, 25, 10…)
  • It displays the local time and the Linux Epoch Time, which is universal (very useful for logs and to detect when there was an issue, for example if your system restarted, your SSH Session would keep latest Epoch captured)
  • No root required
  • Displays recent errors like NFS Timed outs or Memory Read Errors.

Limitations:

  • It only works for Linux, not for Mac or for Windows. Although the idea is to help with Server’s Linux Administration and Troubleshot and Mac and Windows do not have /proc
  • The list of process of the System is read every 30 seconds, to avoid adding much overhead on the System, other info every second

I decided to code name the version 0.7 as “Catalan Republic” to support the dreams and hopes and democratic requests of the Catalans people to become and independent republic.

I created this tool as Open Source and if you want to help I need people to test under different versions of:

  • Atypical Linux distributions

If you are a Cloud Provider and want me to implement the detection of your VMs, so the tool knows that is a instance of the Amazon, Google, Azure, Cloudsigma, Digital Ocean… contact me through my LinkedIn.

Monitoring an Amazon Instance, take a look at the amount of traffic sent and received

Some of the features I’m working on are parsing the logs checking for errors, kernel panics, processed killed due to lack of memory, iscsi disconnects, nfs errors, checking the logs of mysql and Oracle databases to locate errors

Resources for Microservices and Business Domain Solutions for the Cloud Architect / Microservices Architect

First you have to understand that Python, Java and PHP are worlds completely different.

In Python you’ll probably use Flask, and listen to the port you want, inside Docker Container.

In PHP you’ll use a Frameworks like Laravel, or Symfony, or Catalonia Framework (my Framework) :) and a repo or many (as the idea is that the change in one microservice cannot break another it is recommended to have one git repo per Service) and split the requests with the API Gateway and Filters (so /billing/ goes to the right path in the right Server, is like rewriting URLs). You’ll rely in Software to split your microservices. Usually you’ll use Docker, but you have to add a Web Server and any other tools, as the source code is not packet with a Web Server and other Dependencies like it is in Java Spring Boot.

In Java you’ll use Spring Cloud and Spring Boot, and every Service will be auto-contained in its own JAR file, that includes Apache Tomcat and all other Dependencies and normally running inside a Docker. Tcp/Ip listening port will be set at start via command line, or through environment. You’ll have many git repositories, one per each Service.

Using many repos, one per Service, also allows to deploy only that repository and to have better security, with independent deployment tokens.

It is not unlikely that you’ll use one language for some of your Services and another for other, as well as a Database or another, as each Service is owner of their data.

In any case, you will be using CI/CD and your pipeline will be something like this:

  1. Pull the latest code for the Service from the git repository
  2. Compile the code (if needed)
  3. Run the Unit and Integration Tests
  4. Compile the service to an executable artifact (f.e. Java JAR with Tomcat server and other dependencies)
  5. Generate a Machine image with your JAR deployed (for Java. Look at Spotify Docker Plugin to Docker build from Maven), or with Apache, PHP, other dependencies, and the code. Normally will be a Docker image. This image will be immutable. You will probably use Dockerhub.
  6. Machine image will be started. Platform test are run.
  7. If platform tests pass, the service is promoted to the next environment (for example Dev -> Test -> PreProd -> Prod), the exact same machine is started in the next environment and platform tests are repeated.
  8. Before deploying to Production the new Service, I recommend running special Application Tests / Behavior-driven. By this I mean, to conduct tests that really test the functionality of everything, using a real browser and emulating the acts of a user (for example with BeHat, Cucumber or with JMeter).
    I recommend this specially because Microservices are end-points, independent of the implementation, but normally they are API that serve to a whole application. In an Application there are several components, often a change in the Front End can break the application. Imagine a change in Javascript Front End, that results in a call a bit different, for example, with an space before a name. Imagine that the Unit Tests for the Service do not test that, and that was not causing a problem in the old version of the Service and so it will crash when the new Service is deployed. Or another example, imagine that our Service for paying with Visa cards generates IDs for the Payment Gateway, and as a result of the new implementation the IDs generated are returned. With the mocked objects everything works, but when we deploy for real is when we are going to use the actual Bank Payment. This is also why is a good idea to have a PreProduction environment, with PreProduction versions of the actual Services we use (all banks or the GDS for flights/hotel reservation like Galileo or Amadeus have a Test, exactly like Production, Gateway)

If you work with Microsoft .NET, you’ll probably use Azure DevOps.

We IT Engineers, CTOs and Architects, serve the Business. We have to develop the most flexible approaches and enabling the business to release as fast as their need.

Take in count that Microservices is a tool, a pattern. We will use it to bring more flexibility and speed developing, resilience of the services, and speed and independence deploying. However this comes at a cost of complexity.

Microservices is more related to giving flexibility to the Business, and developing according to the Business Domains. Normally oriented to suite an API. If you have an API that is consumed by third party you will have things like independence of Services (if one is down the others will still function), gradual degradation, being able to scale the Services that have more load only, being able to deploy a new version of a Service which is independent of the rest of the Services, etc… the complexity in the technical solution comes from all this resilience, and flexibility.

If your Dev Team is up to 10 Developers or you are writing just a CRUD Web Application, a PoC, or you are an Startup with a critical Time to Market you probably you will not want to use Microservices approach. Is like killing flies with laser cannons. You can use typical Web services approach, do everything in one single Https request, have transactions, a single Database, etc…

But if your team is 100 Developer, like a big eCommerce, you’ll have multiple Teams between 5 and 10 Developers per Business Domain, and you need independence of each Service, having less interdependence. Each Service will own their own Data. That is normally around 5 to 7 tables. Each Service will serve a Business Domain. You’ll benefit from having different technologies for the different needs, however be careful to avoid having Teams with different knowledge that can have hardly rotation and difficult to continue projects when the only 2 or 3 Devs that know that technology leave. Typical benefit scenarios can be having MySql for the Billing Services, but having NoSQL Database for the image catalog, or to store logs of account activity. With Microservices, some services will be calling other Services, often asynchronously, using Queues or Streams, you’ll have Callbacks, Databases for reading, you’ll probably want to have gradual and gracefully failure of your applications, client load balancing, caches and read only databases/in-memory databases… This complexity is in order to protect one Service from the failure of others and to bring it the necessary speed under heavy load.

Here you can find a PDF Document of the typical resources I use for Microservice Projects.

You can also download it from my github repository:

https://github.com/carlesmateo/awesome-microservices

Do you use other solutions that are not listed?. Leave a message. I’ll investigate them and update the Document, to share with the Community.

Update 2020-03-06: I found this very nice article explaining the same. Microservices are not for everybody and not the default option: https://www.theregister.co.uk/AMP/2020/03/04/microservices_last_resort/

Update 2020-03-11: Qcom with 1,600 microservices says that microservices architecture is the las resort: https://www.theregister.co.uk/AMP/2020/03/09/monzo_microservices/

Upgrading my new HP 14-bp060sa

As the company I was working for, Sanmina, has decided to move all the Software Development to Colorado, US, and closing the offices in Bishopstown, Cork, Ireland I found myself with the need to get a new laptop. At work I was using two Dell laptops, one very powerful and heavy equipped with an Intel Xeon processor and 32 GB of RAM. The other a lightweight one that I updated to 32 GB of RAM.

I had an accident around 8 months ago, that got my spine damaged, and so I cannot carry much weight.

My personal laptops at home, in Ireland are a 15″ with 16 GB of RAM, too heavy, and an Acer 11,6″ with 8GB of RAM and SSD (I upgraded it), but unfortunately the screen crashed. I still use it through the HDMI port. My main computer is a tower with a Core i7, 64GB of RAM and a Samsung NVMe SSD drive. And few Raspberrys Pi 4 and 3 :)

I was thinking about what ultra-lightweight laptop to buy, but I wanted to buy it in Barcelona, as I wanted a Catalan keyboard (the layout with the broken ç and accents). I tried by Amazon.es but I have problems to have shipped the Catalan keyboard layout laptops to my address in Ireland.

I was trying to find the best laptop for me.

While I was investigating I found out that none of the laptops in the market were convincing me.

The ones in around 1Kg, which was my initial target, were too big, and lack a proper full size HDMI port and Gigabit Ethernet. Honestly, some models get the HDMI or the Ethernet from an USB 3.1, through an adapter, or have mini-HDMI, many lack the Gigabit port, which is very annoying. Also most of the models come with 8GB of RAM only and were impossible to upgrade. I enrolled my best friend in my quest, in the research, and had the same conclusions.

I don’t want to have to carry adapters with me to just plug to a monitor or projector. I don’t even want to carry the power charger. I want a laptop that can work with me for a complete day, a full work session, without needing to recharge.

So while this investigation was going on, I decided to buy a cheap laptop with a good trade off of weight and cost, in order to be able to work on the coffee. I needed it for writing documents in Google Docs, creating microservices architectures, programming in Java and PHP, and writing articles in my blog. I also decided that this would be my only laptop with Windows, as honestly I missed playing Star Craft 2, and my attempts with Wine and Linux did not success.

Not also, for playing games :) , there are tools that are only available for Windows or for Mac Os X and Windows, like: POSTMAN, Kitematic for managing dockers visually, vSphere…

(Please note, as I reviewed the article I realized that POSTMAN is available for Linux too)

Please note: although I use mainly Linux everywhere (Ubuntu, CentOS, and RedHat mainly) and I contribute to Open Source projects, I do have Windows machines.

I created my Start up in 2004, and I still have Windows Servers, physical machines in a Data Center in Barcelona, and I still have VMs and Instances in Public Clouds with Windows Servers. Also I programmed some tools using Visual Studio and Visual Basic .NET, ASP.NET and C#, but when I needed to do this I found more convenient spawn an instance in Amazon or Azure and pay for its use.

When I created my Start up I offered my infrastructure as a way to get funding too, and I offered VMs with VMWare. I found that having my Mail Servers in VMs was much more convenient for Backups, cloning, to scale up, to avoid disruption and for Disaster and Recovery.

I wanted a cheap laptop that will not make feel bad if transporting it in a daily basis gets a hit and breaks, or that if it rains (and this happens more than often in Ireland) and it breaks is not super-hurtful, or even if it gets stolen. Yes, I’m from a big city, like is Barcelona, Catalonia, and thieves are a real problem. I travel, so I want a laptop decent enough that I can take to travel, and for going for a coffee, coding anything, and I feel comfortable enough that if something happens to it is not the end of the world.

Cork is not a big city, so the options were reduced. I found a laptop that meets my needs.

I got a HP s14-bp060sa for 439€.

It is equipped with a Intel® Core™ i3-6006U (2 GHz, 3 MB cache, 2 cores) , a 500GB SATA HDD, and 4 GB of DDR4 RAM.

The information on HP webpage is really scarce, but checking other pages I was able to see that the motherboard has 2 memory banks, accepting a max of 16GB of RAM.

I saw that there was an slot, unclear if supporting NVMe SSD drives, but supporting M.2 SSD for sure.

So I bought in Amazon 2x8GB and a M.2 500GB drive.

Since I was 5 years old I’ve been upgrading and assembling by myself all the computers. And this is something that I want to keep doing. It keeps me sharp, knowing the new ports, CPUs, and motherboard architectures, and keeps me in contact with the Hardware. All my life I’ve thought that specializing Software Engineers and Systems Engineers, like if computers were something separate, is a mistake, so I push myself to stay up to date of the news in all the fields.

I removed the spinning 500 GB SATA HDD, cause it’s slow and it consumes a lot of energy. With the M.2 SSD the battery last forever.

The interesting part is how I cloned the drive from the Spinning HDD to the new M.2.

I did:

  • Open the computer (see pics below) and Insert the new drive M.2
  • Boot with an USB Linux Rescue distribution (to do that I had to enable Legacy Boot on BIOS and boot with the USB)
  • Use lsblk command to identify the HDD drive, it was easy as it was the one with partitions
  • dd from if=/dev/sda to of=/dev/sdb with status=progress to see live status and speed (around 70MB/s) and estimated time to complete.
  • Please note that the new drive should be bigger or at least have the same number of bytes to avoid problems with the last partition.
  • I removed the HDD drive, this reduces the weight of my laptop by 100 grams
  • Disable Legacy Boot, and boot the computer. Windows started perfectly :)

I found so few information about this model, that I wanted to share the pictures with the Community. Here are the pictures of the upgrade process.

Here you can see the Crucial M.2 SSD installed and the Spinning HDD removed. Yes, I did in a coffee :)
Final step, installing the 2x8GB RAM memory modules

Installing Red Hat Linux in a M.2 that crashes the installer

Few months ago I encountered with a problem with RHEL installer and some of the M.2 drives.

I’ve productized my Product, to be released with M.2 booting SATA drives of 128GB.

The procedure for preparing the Servers (90 and 60 drives, Cold Storage) was based on the installation of RHEL in the M.2 128GB drive. Then the drives are cloned.

Few days before mass delivery the company request to change the booting M.2 drives for others of our own, 512 GB drives.

I’ve tested many different M.2 drives and all of them were slightly different.

Those 512 GB M.2 drives had one problem… Red Hat installer was failing with a python error.

We were running out of time, so I decided to clone directly from the 128GB M.2 working card, with everything installed, to the 512 GB card. Doing that is so easy as booting with a Rescue Linux USB disk, and then doing a dd from the 128GB drive to the 512GB drive.

Booting with a live USB system is important, as Filesystem should not be mounted to prevent corruption when cloning.

Then, the next operation would be booting the 512 GB drive and instructing Linux to claim the additional space.

Here is the procedure for doing it (note, the OS installed in the M.2 was CentOS in this case):

Determine the device that needs to be operated on (this will usually be the boot drive); in this example it is /dev/sdae

# df -h 
Filesystem                             Size  Used Avail Use% Mounted on
/dev/mapper/centos_4602c-root           50G  2.4G   47G   1% /
devtmpfs                                16G     0   16G   0% /dev
tmpfs                                   16G     0   16G   0% /dev/shm
tmpfs                                   16G  395M   16G   3% /run
tmpfs                                   16G     0   16G   0% /sys/fs/cgroup
/dev/sdae1                            1014M  146M  869M  15% /boot
/dev/mapper/centos_4602c-home           57G   33M   57G   1% /home
tmpfs                                  3.2G     0  3.2G   0% /run/user/0
logs                                    68G  7.4M   68G   1% /logs
mysql                                  481G  128K  481G   1% /mysql
N58-C3-D16-P3-S1                       491T  334G  490T   1% /N58-C3-D16-P3-S1

Extend the OS partition using Parted

# parted /dev/sdae
print
resizepart PART_NUMBER END
quit

Where:

  • PART_NUMBER: Is the partition number obtained from the “print” command
  • END: This is the end of the drive; for example, for a 50GB drive, enter 50000

Examining the LVM Partitions

The centos_4602c-root LVM partition is the one we want to extend.

# lsblk /dev/sdae
NAME                          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sdae                           65:224  0   477G  0 disk 
├─sdae1                        65:225  0     1G  0 part /boot
└─sdae2                        65:226  0 475.9G  0 part 
  ├─centos_4602c-root         253:0    0    50G  0 lvm  /
  ├─centos_4602c-swap         253:1    0  11.9G  0 lvm  [SWAP]
  └─centos_4602c-home         253:2    0  56.3G  0 lvm  /home

Using LVM Commands

The following commands will:

  • Display the LVM volumes on the system
  • Resize a volume (device)
  • Re-display the updated LVM volumes
  • Extend the desired LVM partition (lvextend command)
# pvdisplay
  /dev/sdbm: open failed: No medium found
  /dev/sdbn: open failed: No medium found
  /dev/sdbj: open failed: No medium found
  /dev/sdbk: open failed: No medium found
  /dev/sdbl: open failed: No medium found
  --- Physical volume ---
  PV Name               /dev/sdae2
  VG Name               centos_4602c
  PV Size               118.24 GiB / not usable 3.00 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              30269
  Free PE               0
  Allocated PE          30269
  PV UUID               yvHO6t-cYHM-CCCm-2hOO-mJWf-6NUI-zgxzwc
# pvresize /dev/sdae2
  /dev/sdbm: open failed: No medium found
  /dev/sdbn: open failed: No medium found
  /dev/sdbj: open failed: No medium found
  /dev/sdbk: open failed: No medium found
  /dev/sdbl: open failed: No medium found
  Physical volume "/dev/sdae2" changed
  1 physical volume(s) resized or updated / 0 physical volume(s) not resized
# pvdisplay
  /dev/sdbm: open failed: No medium found
  /dev/sdbn: open failed: No medium found
  /dev/sdbj: open failed: No medium found
  /dev/sdbk: open failed: No medium found
  /dev/sdbl: open failed: No medium found
  --- Physical volume ---
  PV Name               /dev/sdae2
  VG Name               centos_4602c
  PV Size               <475.84 GiB / not usable 3.25 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              121813
  Free PE               91544
  Allocated PE          30269
  PV UUID               yvHO6t-cYHM-CCCm-2hOO-mJWf-6NUI-zgxzwc
# vgdisplay
  --- Volume group ---
  VG Name               centos_4602c
  System ID             
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  6
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               3
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               <475.93 GiB
  PE Size               4.00 MiB
  Total PE              121838
  Alloc PE / Size       30269 / <118.24 GiB
  Free  PE / Size       91569 / 357.69 GiB
  VG UUID               ORcp2t-ntwQ-CNSX-NeXL-Udd9-htt9-kLfvRc
# lvextend -l +91569 /dev/centos_4602c/root 
  Size of logical volume centos_4602c/root changed from 50.00 GiB (12800 extents) to <407.69 GiB (104369 extents).
  Logical volume centos_4602c/root successfully resized.

Extend the xfs file system to use the extended space

The xfs file system for the root partition will need to be extended to use the extra space; this is done using the xfs_grow command as shown below.

# xfs_growfs /dev/centos_4602c/root  
meta-data=/dev/mapper/centos_4602c-root isize=512    agcount=4, agsize=3276800 blks
         =                       sectsz=512   attr=2, projid32bit=1          =                       crc=1        finobt=0 spinodes=0 data     =                       bsize=4096   blocks=13107200, imaxpct=25 
         =                       sunit=0      swidth=0 blks 
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1 log      =internal               bsize=4096   blocks=6400, version=2          =                       sectsz=512   sunit=0 blks, lazy-count=1 
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 13107200 to 106873856 

Verify the results

Note that the c-root LVM partition is now 408GB.

# df -h 
Filesystem                             Size  Used Avail Use% Mounted on
/dev/mapper/centos_4602c-root          408G  2.4G  406G   1% /
devtmpfs                                16G     0   16G   0% /dev
tmpfs                                   16G     0   16G   0% /dev/shm
tmpfs                                   16G  395M   16G   3% /run
tmpfs                                   16G     0   16G   0% /sys/fs/cgroup
/dev/sdae1                            1014M  146M  869M  15% /boot
/dev/mapper/centos_4602c-home           57G   33M   57G   1% /home
tmpfs                                  3.2G     0  3.2G   0% /run/user/0
logs                                    68G  7.4M   68G   1% /logs
mysql                                  481G  128K  481G   1% /mysql
N58-C3-D16-P3-S1                       491T  334G  490T   1% /N58-C3-D16-P3-S1

So now we are able to clone directly from one 512GB to another.

You may be interested to take a look to the commands:

growpart
resize2fs
xfs_growfs (from xfsprogs package)

If you want to do this in an instance in Amazon, here is a very good documentation.

A sample forensic post mortem for a iSCSI Initiator (client) that had connectivity problems to the Server

My Team in The States report an issue with a Red Hat iSCSI Initiator having issues connecting to a Volume exported by a ZFS Server.

There is an issue on GitLab.

As I always do when I troubleshot a problem, I create a forensics post-mortem document recording everything I do, so later, others can learn how I fix it, or they can learn the steps I did in order to troubleshoot.

Please note: Some Ip addresses have been manually edited.

2019-08-09 10:20:10 Start of the investigation

I log into the Server, with Ip Address: xxx.yyy.16.30. Is an All-Flash-Array Server with RHEL6.10 and DRAID v.08091350.

Htop shows normal/low activity.

I check the addresses in the iSCSI Initiator (client), to make sure it is connecting to the right Server.

[root@Host-164 ~]# ip addr list 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
    valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:25:90:c5:1e:ea brd ff:ff:ff:ff:ff:ff
    inet xxx.yyy.13.164/16 brd xxx.yyy.255.255 scope global eno1
    valid_lft forever preferred_lft forever
    inet6 fe80::225:90ff:fec5:1eea/64 scope link
    valid_lft forever preferred_lft forever
3: eno2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 00:25:90:c5:1e:eb brd ff:ff:ff:ff:ff:ff 
4: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 24:8a:07:a4:94:9c brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.164/24 brd 192.168.100.255 scope global enp3s0f0
    valid_lft forever preferred_lft forever
    inet6 fe80::268a:7ff:fea4:949c/64 scope link
    valid_lft forever preferred_lft forever 
5: enp3s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 24:8a:07:a4:94:9d brd ff:ff:ff:ff:ff:ff
    inet 192.168.200.164/24 brd 192.168.200.255 scope global enp3s0f1
    valid_lft forever preferred_lft forever inet6 fe80::268a:7ff:fea4:949d/64 scope link
    valid_lft forever preferred_lft forever                                                                                                       
                                                                                                                                

I see the luns on the host, connecting to the 10Gbps of the Server:

[root@Host-164 ~]# iscsiadm -m session
 tcp: [10] 192.168.100.30:3260,1 iqn.2003-01.org.linux-iscsi:vol4 (non-flash)
 tcp: [11] 192.168.100.30:3260,1 iqn.2003-01.org.linux-iscsi:vol5 (non-flash)
 tcp: [7] 192.168.100.30:3260,1 iqn.2003-01.org.linux-iscsi:vol1 (non-flash)
 tcp: [8] 192.168.100.30:3260,1 iqn.2003-01.org.linux-iscsi:vol2 (non-flash)
 tcp: [9] 192.168.100.30:3260,1 iqn.2003-01.org.linux-iscsi:vol3 (non-flash)

Finding the misteries…

Executing cat /proc/partitions is a bit strange respect mount:

[root@Host-164 ~]# cat /proc/partitions
 major minor #blocks name
 8  0 125034840 sda
 8  1 512000 sda1
 8  2 124521472 sda2
 253 0 12505088 dm-0
 253 1 112013312 dm-1
 8 32 104857600 sdc
 8 16 104857600 sdb
 8 48 104857600 sdd
 8 64 104857600 sde
 8 80 104857600 sdf

As mount has this:

/dev/sdg1 on /mnt/large type ext4 (ro,relatime,seclabel,data=ordered)

Lsblk shows that /dev/sdg is not present:

[root@Host-164 ~]# lsblk
 NAME
 MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
 sda 8:0 0 119.2G 0 disk
 ├─sda1 8:1 0 500M 0 part /boot
 └─sda2 8:2 0 118.8G 0 part
  ├─rhel-swap 253:0 0 11.9G 0 lvm [SWAP]
  └─rhel-root 253:1 0 106.8G 0 lvm /
 sdb 8:16 0 100G 0 disk
 sdc 8:32 0 100G 0 disk
 sdd 8:48 0 100G 0 disk
 sde 8:64 0 100G 0 disk
 sdf 8:80 0 100G 0 disk

And as expected:

[root@Host-164 ~]# ls -al /mnt/large
 ls: reading directory /mnt/large: Input/output error
 total 0

I see that the Volumes appear to not having being partitioned:

[root@Host-164 ~]# fdisk /dev/sdf
 Welcome to fdisk (util-linux 2.23.2).
 Changes will remain in memory only, until you decide to write them.
 Be careful before using the write command.
 Device does not contain a recognized partition table
 Building a new DOS disklabel with disk identifier 0xddf99f40.
 Command (m for help): p
 Disk /dev/sdf: 107.4 GB, 107374182400 bytes, 209715200 sectors
 Units = sectors of 1 * 512 = 512 bytes
 Sector size (logical/physical): 512 bytes / 512 bytes
 I/O size (minimum/optimal): 512 bytes / 512 bytes
 Disk label type: dos
 Disk identifier: 0xddf99f40
 Device Boot
 Start
 End
 Blocks Id System
 Command (m for help): q

I create a partition and format with ext2

[root@Host-164 ~]# mke2fs /dev/sdb1
 mke2fs 1.42.9 (28-Dec-2013)
 Filesystem label=
 OS type: Linux
 Block size=4096 (log=2)
 Fragment size=4096 (log=2)
 Stride=0 blocks, Stripe width=0 blocks
 6553600 inodes, 26214144 blocks
 1310707 blocks (5.00%) reserved for the super user
 First data block=0
 Maximum filesystem blocks=4294967296
 800 block groups
 32768 blocks per group, 32768 fragments per group
 8192 inodes per group
 Superblock backups stored on blocks:
 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
 4096000, 7962624, 11239424, 20480000, 23887872
 Allocating group tables: done
 Writing inode tables: done
 Writing superblocks and filesystem accounting information: done

I mount:

[root@Host-164 ~]# mount /dev/sdb1 /mnt/vol1

I fill the volume from the client, and it works. I check the activity in the Server with iostat and there are more MB/s written to the Server’s drives than actually speed copying in the client.

I completely fill 100GB but speed is slow. We are working on a 10Gbps Network so I expected more speed.

I check the connections to the Server:

[root@obs4602-1810 ~]# netstat | grep -v "unix"
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State      
tcp        0      0 192.168.10.10:iscsi-target  192.168.10.180:55300        ESTABLISHED
tcp        0      0 192.168.10.10:iscsi-target  192.168.10.180:55298        ESTABLISHED
tcp        0      0 xxx.yyy.18.10:ssh            xxx.yyy.12.154:57137         ESTABLISHED
tcp        0      0 192.168.10.10:iscsi-target  192.168.10.180:55304        ESTABLISHED
tcp        0      0 192.168.10.10:iscsi-target  192.168.10.180:55301        ESTABLISHED
tcp        0      0 192.168.10.10:iscsi-target  192.168.10.180:55306        ESTABLISHED
tcp        0      0 xxx.yyy.18.10:ssh            xxx.yyy.12.154:56395         ESTABLISHED
tcp        0      0 xxx.yyy.18.10:ssh            xxx.yyy.14.52:57330          ESTABLISHED
tcp        0      0 192.168.10.10:iscsi-target  192.168.10.180:55296        ESTABLISHED
tcp        0      0 192.168.10.10:iscsi-target  192.168.10.180:55305        ESTABLISHED
tcp        0      0 xxx.yyy.18.10:ssh            xxx.yyy.12.154:57133         ESTABLISHED
tcp        0      0 192.168.10.10:iscsi-target  192.168.10.180:55303        ESTABLISHED
tcp        0      0 192.168.10.10:iscsi-target  192.168.10.180:55299        ESTABLISHED
tcp        0      0 192.168.10.10:iscsi-target  192.168.10.176:57542        ESTABLISHED
tcp        0      0 192.168.10.10:iscsi-target  192.168.10.180:55302        ESTABLISHED

I see many connections from Host 180, I check that and another member of the Team is using that client to test with vdbench against the Server.

This explains the slower speed I was getting.

Conclusions

  1. There was a local problem on the Host. The problems with the disconnection seem to be related to a connection that was lost (sdg). All that information was written to iSCSI buffer, not to the Server. In fact, that volume was mapped in the system with another letter, sdg was not in use.
  2. Speed was slow due to another client pushing Data to the Server too
  3. Windows clients with auto reconnect option are not reporting timeout reports while in Red Hat clients iSCSI connection timeouts. It should be increased

2020-03-10 22:16 IST TIP: At that time we were using Google suite and Skype to communicate internally with the different members across the world. If we had used a tool like Slack, and we had a channel like #engineering for example or #sanjoselab, then I could have paged and asked “Is somebody using obs4602-1810?

Creating a VM for compiling ZFS with RHEL6.10

As you know I created the DRAID project, based in ZFS.

One of our customers wanted a special custom version for their RHEL6.10 installation with a custom Kernel.

This post describes how to compile and install ZFS 7.x for RHEL6.

First create a VM with RHEL6.10. Myself I used Virtual Box on Ubuntu.

If you need to install a Custom Kernel matching the destination Servers, do it.

Download the source code from ZFS for Linux.

install the following packages which are required by zfs compiler:

sudo yum groupinstall "Development Tools"
sudo yum install autoconf automake libtool wget libtirpc-devel rpm-build
sudo yum install zlib-devel libuuid-devel libattr-devel libblkid-devel libselinux-devel libudev-devel
sudo yum install parted lsscsi ksh openssl-devel elfutils-libelf-develsudo yum install kernel-devel-$(uname -r)

steps to compile the code:1- make sure  the zfs file exists under zfs/contrib/initramfs/scripts/local-top/

if not exists, create a file called zfs  under zfs/contrib/initramfs/scripts/local-top/  and add the following to that file:

#!/bin/sh
PREREQ=”mdadm mdrun multipath”

prereqs()
{
       echo “$PREREQ”
}

case $1 in
# get pre-requisites
prereqs)
       prereqs
       exit 0
       ;;
esac


#
# Helper functions
#
message()
{
       if [ -x /bin/plymouth ] && plymouth –ping; then
               plymouth message –text=”$@”
       else
               echo “$@” >&2
       fi
       return 0
}

udev_settle()
{
       # Wait for udev to be ready, see https://launchpad.net/bugs/85640
       if [ -x /sbin/udevadm ]; then
               /sbin/udevadm settle –timeout=30
       elif [ -x /sbin/udevsettle ]; then
               /sbin/udevsettle –timeout=30
       fi
       return 0
}


activate_vg()
{
       # Sanity checks
       if [ ! -x /sbin/lvm ]; then
               [ “$quiet” != “y” ] && message “lvm is not available”
               return 1
       fi

       # Detect and activate available volume groups
       /sbin/lvm vgscan
       /sbin/lvm vgchange -a y –sysinit
       return $?
}

udev_settle
activate_vg

exit 0

make the created zfs file executable:

chmod +x  zfs/contrib/initramfs/scripts/local-top/zfs

2-  inside  draid-zfs-2019-05-09 folder, execute the following commands:execute Auto generate script:

./autogen.sh

execute configuration script:

./configure

Please note we use this specific configuration for bettter results:

./configure –disable-pyzfs –with-spec=redhat

create rpms:

make rpm

remove all test rpms:

rm zfs-test*.rpm

3- install all created rpms

yum install *x86_64* -y

4- verify that zfs is been installed

zfs

this command will display zfs help. 

Another interesting trick I instructed my Team to do is to add a version number to zfs, with a parameter -v or –version.

So if you want to do the same, you have to edit:

zfs/cmd/zfs/zfs_main.c

Under:

cmdname = argv[1];

In my code is line 7926, then add:

/* DRAIDTEAM - added new command to display zfs version*/
if ((strcmp(cmdname, "-v") == 0) || (strcmp(cmdname, "--version") == 0)) {
    (void) fprintf(stdout, "0.7.0_DRAID-1.2.9.08021755\n");
    return (0);
}

You can check the Kernel Module info by using modinfo zfs, but I found it handy to allow to just do:

zfs -v

A handy trick command line to get the usages of our Python Methods in the code

We all use powerful code analysis tool, but sometimes you’re presented with a problem and you have just… the terminal.

This Bash code is handy.

grep "def " /home/carles/code/gitlab/cloud/terraform/src/scale/lib/iscsi.py | tr "()" "  " | awk '{ print $2; }' |  grep -v "__init" | sort > ./function_names_iscsi.txt

So this basically will get all the methods (“def ” whatever), strip the parenthesis with tr, and get the second column with awk, so basically the method name, sort it and write it to the file.

Then I will cd to the src directory and execute the seconds part:

cd /home/carles/code/gitlab/cloud/terraform/src/
for fname in $(cat ~/function_names_iscsi.txt); do printf "%s: %s\n" "$fname" "$(grep -r $fname *|grep -v 'def ' -c)"; done > ~/functions_being_used.txt

That will produce a nice list with the number of times of the method being called, in the form of:

method_name: occurrences

That’s the equivalent to doing Find Usages is PyCharm.

It’s easy to identify dead code then, with method_name: 0.

You can also run this to your Jenkins to warn when there is Dead Code in your repository.